From 2ae243fd0313872f4c5b026f877d8d2366990b4a Mon Sep 17 00:00:00 2001 From: Ricardo Dahis Date: Mon, 24 Aug 2026 16:28:02 +1000 Subject: [PATCH 1/6] fix(world_wb_mides): remove phantom models, unpin staging project, widen partitions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three defects found while auditing MiDES against production BigQuery. 1. 40 of the 49 model files were empty (29 zero-byte) or comment-only (11 containing just `-- update_data`). dbt silently skips the zero-byte ones, but registers the 11 comment-only files as real nodes that compile to `-- update_data` — invalid as `create or replace table X as (...)`, so `dbt run --select world_wb_mides` fails on every one of them. None is referenced by schema.yml or any ref(), and none has ever existed as a table in dev or prod. Removed all 40; the model count drops 1087 -> 1047 and the dataset compiles clean. 2. 36 references hardcoded `basedosdados-staging` instead of going through set_datalake_project — every aux_* lookup plus raw_rsp_mg, raw_empenho_pr and raw_empenho_pb. The macro resolves prod to basedosdados-staging, so prod behaviour is unchanged by construction; what this fixes is the dev target, which was reading *production* staging for those tables and so never compared like with like. Compiled empenho now resolves 34 refs to basedosdados-dev and 2 to basedosdados (the br_bd_diretorios_brasil joins, which are correct as full paths). 3. Partition ranges did not cover the data. Measured against basedosdados.world_wb_mides on 2026-08-24: empenho spans 1989-2024 but declared 1994-2024; liquidacao spans 1992-2024, declared 1995-2024; and pagamento spans 1994-2024 while declared 1996-2022, so DF 2023 and SC 2023-2024 rows land in __UNPARTITIONED__ and lose partition pruning. Widened to latest+5 per the house convention: 1989-2029, 1992-2029, 1994-2029, and the three licitacao tables 2009-2026. No data is rebuilt by this commit; it makes the next materialisation correct. --- .../world_wb_mides__aux_funcao.sql | 0 .../world_wb_mides__aux_municipio_ce.sql | 0 .../world_wb_mides__aux_municipio_pb.sql | 0 .../world_wb_mides__aux_municipio_pe.sql | 0 .../world_wb_mides__aux_municipio_sp.sql | 0 .../world_wb_mides__aux_orgao_rs.sql | 0 .../world_wb_mides__aux_subfuncao.sql | 0 .../world_wb_mides__empenho.sql | 62 ++++++++++++------- .../world_wb_mides__licitacao.sql | 2 +- .../world_wb_mides__licitacao_item.sql | 2 +- ...world_wb_mides__licitacao_participante.sql | 2 +- .../world_wb_mides__liquidacao.sql | 43 ++++++++----- .../world_wb_mides__pagamento.sql | 51 ++++++++++----- .../world_wb_mides__raw_anulacao_ce.sql | 0 .../world_wb_mides__raw_anulacao_rj.sql | 2 - ...wb_mides__raw_despesa_ato_rj_municipio.sql | 2 - ...rld_wb_mides__raw_despesa_rj_municipio.sql | 2 - .../world_wb_mides__raw_despesa_rs.sql | 0 .../world_wb_mides__raw_despesa_sp.sql | 0 ...rld_wb_mides__raw_despesa_sp_municipio.sql | 2 - .../world_wb_mides__raw_empenho_ce.sql | 0 .../world_wb_mides__raw_empenho_df.sql | 2 - .../world_wb_mides__raw_empenho_mg.sql | 0 .../world_wb_mides__raw_empenho_pb.sql | 0 .../world_wb_mides__raw_empenho_pe.sql | 0 .../world_wb_mides__raw_empenho_pr.sql | 0 .../world_wb_mides__raw_empenho_rj.sql | 2 - .../world_wb_mides__raw_empenho_sc.sql | 2 - .../world_wb_mides__raw_estorno_pb.sql | 0 .../world_wb_mides__raw_liquidacao_ce.sql | 0 .../world_wb_mides__raw_liquidacao_df.sql | 2 - .../world_wb_mides__raw_liquidacao_mg.sql | 0 .../world_wb_mides__raw_liquidacao_pb.sql | 0 .../world_wb_mides__raw_liquidacao_pe.sql | 0 .../world_wb_mides__raw_liquidacao_pr.sql | 0 .../world_wb_mides__raw_liquidacao_rj.sql | 2 - .../world_wb_mides__raw_pagamento_ce.sql | 0 .../world_wb_mides__raw_pagamento_df.sql | 2 - .../world_wb_mides__raw_pagamento_mg.sql | 0 .../world_wb_mides__raw_pagamento_pb.sql | 0 .../world_wb_mides__raw_pagamento_pe.sql | 0 .../world_wb_mides__raw_pagamento_pr.sql | 0 .../world_wb_mides__raw_pagamento_rj.sql | 2 - .../world_wb_mides__raw_resumo_pe.sql | 0 .../world_wb_mides__raw_rsp_mg.sql | 0 .../world_wb_mides__raw_rsp_sp.sql | 0 46 files changed, 108 insertions(+), 76 deletions(-) delete mode 100644 models/world_wb_mides/world_wb_mides__aux_funcao.sql delete mode 100644 models/world_wb_mides/world_wb_mides__aux_municipio_ce.sql delete mode 100644 models/world_wb_mides/world_wb_mides__aux_municipio_pb.sql delete mode 100644 models/world_wb_mides/world_wb_mides__aux_municipio_pe.sql delete mode 100644 models/world_wb_mides/world_wb_mides__aux_municipio_sp.sql delete mode 100644 models/world_wb_mides/world_wb_mides__aux_orgao_rs.sql delete mode 100644 models/world_wb_mides/world_wb_mides__aux_subfuncao.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_anulacao_ce.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_anulacao_rj.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_despesa_ato_rj_municipio.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_despesa_rj_municipio.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_despesa_rs.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_despesa_sp.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_despesa_sp_municipio.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_empenho_ce.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_empenho_df.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_empenho_mg.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_empenho_pb.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_empenho_pe.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_empenho_pr.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_empenho_rj.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_empenho_sc.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_estorno_pb.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_liquidacao_ce.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_liquidacao_df.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_liquidacao_mg.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_liquidacao_pb.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_liquidacao_pe.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_liquidacao_pr.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_liquidacao_rj.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_pagamento_ce.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_pagamento_df.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_pagamento_mg.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_pagamento_pb.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_pagamento_pe.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_pagamento_pr.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_pagamento_rj.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_resumo_pe.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_rsp_mg.sql delete mode 100644 models/world_wb_mides/world_wb_mides__raw_rsp_sp.sql diff --git a/models/world_wb_mides/world_wb_mides__aux_funcao.sql b/models/world_wb_mides/world_wb_mides__aux_funcao.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__aux_municipio_ce.sql b/models/world_wb_mides/world_wb_mides__aux_municipio_ce.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__aux_municipio_pb.sql b/models/world_wb_mides/world_wb_mides__aux_municipio_pb.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__aux_municipio_pe.sql b/models/world_wb_mides/world_wb_mides__aux_municipio_pe.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__aux_municipio_sp.sql b/models/world_wb_mides/world_wb_mides__aux_municipio_sp.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__aux_orgao_rs.sql b/models/world_wb_mides/world_wb_mides__aux_orgao_rs.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__aux_subfuncao.sql b/models/world_wb_mides/world_wb_mides__aux_subfuncao.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__empenho.sql b/models/world_wb_mides/world_wb_mides__empenho.sql index 9ab254c772..21fec9a207 100644 --- a/models/world_wb_mides/world_wb_mides__empenho.sql +++ b/models/world_wb_mides/world_wb_mides__empenho.sql @@ -6,7 +6,7 @@ partition_by={ "field": "ano", "data_type": "int64", - "range": {"start": 1994, "end": 2024, "interval": 1}, + "range": {"start": 1989, "end": 2029, "interval": 1}, }, cluster_by=["mes", "sigla_uf"], labels={"tema": "economia"}, @@ -424,13 +424,16 @@ from from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_pb") }} e left join - `basedosdados-staging.world_wb_mides_staging.aux_municipio_pb` m - on e.cd_ugestora = safe_cast(m.id_unidade_gestora as string) + {{ + set_datalake_project( + "world_wb_mides_staging.aux_municipio_pb" + ) + }} m on e.cd_ugestora = safe_cast(m.id_unidade_gestora as string) left join - `basedosdados-staging.world_wb_mides_staging.aux_funcao` f + {{ set_datalake_project("world_wb_mides_staging.aux_funcao") }} f on e.de_funcao = f.nome_funcao left join - `basedosdados-staging.world_wb_mides_staging.aux_subfuncao` sf + {{ set_datalake_project("world_wb_mides_staging.aux_subfuncao") }} sf on e.de_subfuncao = sf.nome_subfuncao ), anulacao_pb as ( @@ -450,8 +453,11 @@ from from {{ set_datalake_project("world_wb_mides_staging.raw_estorno_pb") }} a left join - `basedosdados-staging.world_wb_mides_staging.aux_municipio_pb` m - on a.cd_ugestora = safe_cast(m.id_unidade_gestora as string) + {{ + set_datalake_project( + "world_wb_mides_staging.aux_municipio_pb" + ) + }} m on a.cd_ugestora = safe_cast(m.id_unidade_gestora as string) group by 1 ), frequencia_pb as ( @@ -855,10 +861,13 @@ from from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_pe") }} e left join - `basedosdados-staging.world_wb_mides_staging.aux_municipio_pe` m - on e.nomeunidadegestora = m.nomeunidadegestora + {{ + set_datalake_project( + "world_wb_mides_staging.aux_municipio_pe" + ) + }} m on e.nomeunidadegestora = m.nomeunidadegestora left join - `basedosdados-staging.world_wb_mides_staging.aux_funcao` fun + {{ set_datalake_project("world_wb_mides_staging.aux_funcao") }} fun on upper( trim( replace( @@ -872,7 +881,7 @@ from ) = upper(nome_funcao) left join - `basedosdados-staging.world_wb_mides_staging.aux_subfuncao` sub + {{ set_datalake_project("world_wb_mides_staging.aux_subfuncao") }} sub on upper(trim(e.subfuncao)) = upper(nome_subfuncao) ), empenho_pr as ( @@ -969,7 +978,8 @@ from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} as c left join - `basedosdados-staging.world_wb_mides_staging.aux_orgao_rs` as a + {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} + as a on c.cd_orgao = a.cd_orgao left join `basedosdados.br_bd_diretorios_brasil.municipio` m @@ -1021,7 +1031,8 @@ from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} as c left join - `basedosdados-staging.world_wb_mides_staging.aux_orgao_rs` as a + {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} + as a on c.cd_orgao = a.cd_orgao left join `basedosdados.br_bd_diretorios_brasil.municipio` m @@ -1265,13 +1276,16 @@ from from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_sp") }} e left join - `basedosdados-staging.world_wb_mides_staging.aux_municipio_sp` m - on m.ds_orgao = e.ds_orgao + {{ + set_datalake_project( + "world_wb_mides_staging.aux_municipio_sp" + ) + }} m on m.ds_orgao = e.ds_orgao left join - `basedosdados-staging.world_wb_mides_staging.aux_funcao` + {{ set_datalake_project("world_wb_mides_staging.aux_funcao") }} on ds_funcao_governo = upper(nome_funcao) left join - `basedosdados-staging.world_wb_mides_staging.aux_subfuncao` + {{ set_datalake_project("world_wb_mides_staging.aux_subfuncao") }} on ds_subfuncao_governo = upper(nome_subfuncao) where tp_despesa = 'Empenhado' ), @@ -1300,8 +1314,11 @@ from from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_sp") }} a left join - `basedosdados-staging.world_wb_mides_staging.aux_municipio_sp` m - on m.ds_orgao = a.ds_orgao + {{ + set_datalake_project( + "world_wb_mides_staging.aux_municipio_sp" + ) + }} m on m.ds_orgao = a.ds_orgao where tp_despesa = 'Anulação' group by 1 ), @@ -1324,8 +1341,11 @@ from from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_sp") }} r left join - `basedosdados-staging.world_wb_mides_staging.aux_municipio_sp` m - on m.ds_orgao = r.ds_orgao + {{ + set_datalake_project( + "world_wb_mides_staging.aux_municipio_sp" + ) + }} m on m.ds_orgao = r.ds_orgao where tp_despesa = 'Reforço' group by 1 ), diff --git a/models/world_wb_mides/world_wb_mides__licitacao.sql b/models/world_wb_mides/world_wb_mides__licitacao.sql index 3731eaee1c..9ca8029e9d 100644 --- a/models/world_wb_mides/world_wb_mides__licitacao.sql +++ b/models/world_wb_mides/world_wb_mides__licitacao.sql @@ -6,7 +6,7 @@ partition_by={ "field": "ano", "data_type": "int64", - "range": {"start": 2009, "end": 2021, "interval": 1}, + "range": {"start": 2009, "end": 2026, "interval": 1}, }, cluster_by=["mes", "sigla_uf"], labels={"tema": "economia"}, diff --git a/models/world_wb_mides/world_wb_mides__licitacao_item.sql b/models/world_wb_mides/world_wb_mides__licitacao_item.sql index a3ceeaf2ea..2ba7256ad9 100644 --- a/models/world_wb_mides/world_wb_mides__licitacao_item.sql +++ b/models/world_wb_mides/world_wb_mides__licitacao_item.sql @@ -6,7 +6,7 @@ partition_by={ "field": "ano", "data_type": "int64", - "range": {"start": 2009, "end": 2021, "interval": 1}, + "range": {"start": 2009, "end": 2026, "interval": 1}, }, cluster_by=["sigla_uf"], labels={"tema": "economia"}, diff --git a/models/world_wb_mides/world_wb_mides__licitacao_participante.sql b/models/world_wb_mides/world_wb_mides__licitacao_participante.sql index 2daa56f9ca..6e2b9cdca6 100644 --- a/models/world_wb_mides/world_wb_mides__licitacao_participante.sql +++ b/models/world_wb_mides/world_wb_mides__licitacao_participante.sql @@ -6,7 +6,7 @@ partition_by={ "field": "ano", "data_type": "int64", - "range": {"start": 2009, "end": 2021, "interval": 1}, + "range": {"start": 2009, "end": 2026, "interval": 1}, }, cluster_by=["sigla_uf"], labels={"tema": "economia"}, diff --git a/models/world_wb_mides/world_wb_mides__liquidacao.sql b/models/world_wb_mides/world_wb_mides__liquidacao.sql index 96ff7b94e7..284fa88f3f 100644 --- a/models/world_wb_mides/world_wb_mides__liquidacao.sql +++ b/models/world_wb_mides/world_wb_mides__liquidacao.sql @@ -6,7 +6,7 @@ partition_by={ "field": "ano", "data_type": "int64", - "range": {"start": 1995, "end": 2024, "interval": 1}, + "range": {"start": 1992, "end": 2029, "interval": 1}, }, cluster_by=["mes", "sigla_uf"], labels={"tema": "economia"}, @@ -89,8 +89,11 @@ from ) }} l left join - `basedosdados-staging.world_wb_mides_staging.aux_municipio_ce` m - on l.codigo_municipio = m.codigo_municipio + {{ + set_datalake_project( + "world_wb_mides_staging.aux_municipio_ce" + ) + }} m on l.codigo_municipio = m.codigo_municipio ), liquidacao_mg as ( select @@ -173,7 +176,7 @@ from ) }} as l left join - `basedosdados-staging.world_wb_mides_staging.raw_rsp_mg` as r + {{ set_datalake_project("world_wb_mides_staging.raw_rsp_mg") }} as r on l.id_rsp = r.id_rsp ), liquidacao_pb as ( @@ -223,8 +226,11 @@ from ) }} l left join - `basedosdados-staging.world_wb_mides_staging.aux_municipio_pb` m - on l.cd_ugestora = safe_cast(m.id_unidade_gestora as string) + {{ + set_datalake_project( + "world_wb_mides_staging.aux_municipio_pb" + ) + }} m on l.cd_ugestora = safe_cast(m.id_unidade_gestora as string) ), liquidacao_pr as ( select @@ -266,7 +272,7 @@ from basedosdados.br_bd_diretorios_brasil.municipio m on cdibge = id_municipio_6 left join - `basedosdados-staging.world_wb_mides_staging.raw_empenho_pr` e + {{ set_datalake_project("world_wb_mides_staging.raw_empenho_pr") }} e on l.idempenho = e.idempenho ), liquidacao_pe as ( @@ -298,7 +304,11 @@ from ) }} l left join - `basedosdados-staging.world_wb_mides_staging.aux_municipio_pe` m + {{ + set_datalake_project( + "world_wb_mides_staging.aux_municipio_pe" + ) + }} m on l.id_unidade_gestora = safe_cast(m.id_unidadegestora as string) ), liquidado_rs as ( @@ -353,7 +363,8 @@ from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} as c left join - `basedosdados-staging.world_wb_mides_staging.aux_orgao_rs` as a + {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} + as a on c.cd_orgao = a.cd_orgao left join `basedosdados.br_bd_diretorios_brasil.municipio` m @@ -379,7 +390,8 @@ from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} as c left join - `basedosdados-staging.world_wb_mides_staging.aux_orgao_rs` as a + {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} + as a on c.cd_orgao = a.cd_orgao left join `basedosdados.br_bd_diretorios_brasil.municipio` m @@ -541,13 +553,16 @@ from from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_sp") }} e left join - `basedosdados-staging.world_wb_mides_staging.aux_municipio_sp` m - on m.ds_orgao = e.ds_orgao + {{ + set_datalake_project( + "world_wb_mides_staging.aux_municipio_sp" + ) + }} m on m.ds_orgao = e.ds_orgao left join - `basedosdados-staging.world_wb_mides_staging.aux_funcao` + {{ set_datalake_project("world_wb_mides_staging.aux_funcao") }} on ds_funcao_governo = upper(nome_funcao) left join - `basedosdados-staging.world_wb_mides_staging.aux_subfuncao` + {{ set_datalake_project("world_wb_mides_staging.aux_subfuncao") }} on ds_subfuncao_governo = upper(nome_subfuncao) where tp_despesa = 'Valor Liquidado' ), diff --git a/models/world_wb_mides/world_wb_mides__pagamento.sql b/models/world_wb_mides/world_wb_mides__pagamento.sql index f843ba06d0..44bf91a58f 100644 --- a/models/world_wb_mides/world_wb_mides__pagamento.sql +++ b/models/world_wb_mides/world_wb_mides__pagamento.sql @@ -6,7 +6,7 @@ partition_by={ "field": "ano", "data_type": "int64", - "range": {"start": 1996, "end": 2022, "interval": 1}, + "range": {"start": 1994, "end": 2029, "interval": 1}, }, cluster_by=["mes", "sigla_uf"], labels={"tema": "economia"}, @@ -68,8 +68,11 @@ from from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_ce") }} e left join - `basedosdados-staging.world_wb_mides_staging.aux_municipio_ce` m - on e.codigo_municipio = m.codigo_municipio + {{ + set_datalake_project( + "world_wb_mides_staging.aux_municipio_ce" + ) + }} m on e.codigo_municipio = m.codigo_municipio ), pago_ce as ( select @@ -144,8 +147,11 @@ from ) }} p left join - `basedosdados-staging.world_wb_mides_staging.aux_municipio_ce` m - on p.codigo_municipio = m.codigo_municipio + {{ + set_datalake_project( + "world_wb_mides_staging.aux_municipio_ce" + ) + }} m on p.codigo_municipio = m.codigo_municipio ), frequencia_ce as ( select id_pagamento_bd, count(id_pagamento_bd) as frequencia_id @@ -309,7 +315,7 @@ from ) }} as p left join - `basedosdados-staging.world_wb_mides_staging.raw_rsp_mg` as r + {{ set_datalake_project("world_wb_mides_staging.raw_rsp_mg") }} as r on p.id_rsp = r.id_rsp ), pago_pb as ( @@ -382,13 +388,17 @@ from ) }} p left join - `basedosdados-staging.world_wb_mides_staging.raw_empenho_pb` e + {{ set_datalake_project("world_wb_mides_staging.raw_empenho_pb") }} e on p.nu_empenho = e.nu_empenho and p.cd_ugestora = e.cd_ugestora and p.de_uorcamentaria = e.de_uorcamentaria and p.dt_ano = e.dt_ano left join - `basedosdados-staging.world_wb_mides_staging.aux_municipio_pb` m + {{ + set_datalake_project( + "world_wb_mides_staging.aux_municipio_pb" + ) + }} m on safe_cast(e.cd_ugestora as string) = safe_cast(m.id_unidade_gestora as string) ), @@ -478,7 +488,11 @@ from ) }} p inner join - `basedosdados-staging.world_wb_mides_staging.aux_municipio_pe` m + {{ + set_datalake_project( + "world_wb_mides_staging.aux_municipio_pe" + ) + }} m on safe_cast(p.id_unidade_gestora as string) = safe_cast(m.id_unidadegestora as string) ), @@ -524,7 +538,7 @@ from ) }} p left join - `basedosdados-staging.world_wb_mides_staging.raw_empenho_pr` e + {{ set_datalake_project("world_wb_mides_staging.raw_empenho_pr") }} e on p.idempenho = e.idempenho left join basedosdados.br_bd_diretorios_brasil.municipio m @@ -602,7 +616,8 @@ from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} as c left join - `basedosdados-staging.world_wb_mides_staging.aux_orgao_rs` as a + {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} + as a on c.cd_orgao = a.cd_orgao left join `basedosdados.br_bd_diretorios_brasil.municipio` m @@ -649,7 +664,8 @@ from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} as c left join - `basedosdados-staging.world_wb_mides_staging.aux_orgao_rs` as a + {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} + as a on c.cd_orgao = a.cd_orgao left join `basedosdados.br_bd_diretorios_brasil.municipio` m @@ -910,13 +926,16 @@ from from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_sp") }} e left join - `basedosdados-staging.world_wb_mides_staging.aux_municipio_sp` m - on m.ds_orgao = e.ds_orgao + {{ + set_datalake_project( + "world_wb_mides_staging.aux_municipio_sp" + ) + }} m on m.ds_orgao = e.ds_orgao left join - `basedosdados-staging.world_wb_mides_staging.aux_funcao` + {{ set_datalake_project("world_wb_mides_staging.aux_funcao") }} on ds_funcao_governo = upper(nome_funcao) left join - `basedosdados-staging.world_wb_mides_staging.aux_subfuncao` + {{ set_datalake_project("world_wb_mides_staging.aux_subfuncao") }} on ds_subfuncao_governo = upper(nome_subfuncao) where tp_despesa = 'Valor Pago' ), diff --git a/models/world_wb_mides/world_wb_mides__raw_anulacao_ce.sql b/models/world_wb_mides/world_wb_mides__raw_anulacao_ce.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_anulacao_rj.sql b/models/world_wb_mides/world_wb_mides__raw_anulacao_rj.sql deleted file mode 100644 index 53d94721e8..0000000000 --- a/models/world_wb_mides/world_wb_mides__raw_anulacao_rj.sql +++ /dev/null @@ -1,2 +0,0 @@ --- update_data - diff --git a/models/world_wb_mides/world_wb_mides__raw_despesa_ato_rj_municipio.sql b/models/world_wb_mides/world_wb_mides__raw_despesa_ato_rj_municipio.sql deleted file mode 100644 index 53d94721e8..0000000000 --- a/models/world_wb_mides/world_wb_mides__raw_despesa_ato_rj_municipio.sql +++ /dev/null @@ -1,2 +0,0 @@ --- update_data - diff --git a/models/world_wb_mides/world_wb_mides__raw_despesa_rj_municipio.sql b/models/world_wb_mides/world_wb_mides__raw_despesa_rj_municipio.sql deleted file mode 100644 index 53d94721e8..0000000000 --- a/models/world_wb_mides/world_wb_mides__raw_despesa_rj_municipio.sql +++ /dev/null @@ -1,2 +0,0 @@ --- update_data - diff --git a/models/world_wb_mides/world_wb_mides__raw_despesa_rs.sql b/models/world_wb_mides/world_wb_mides__raw_despesa_rs.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_despesa_sp.sql b/models/world_wb_mides/world_wb_mides__raw_despesa_sp.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_despesa_sp_municipio.sql b/models/world_wb_mides/world_wb_mides__raw_despesa_sp_municipio.sql deleted file mode 100644 index 53d94721e8..0000000000 --- a/models/world_wb_mides/world_wb_mides__raw_despesa_sp_municipio.sql +++ /dev/null @@ -1,2 +0,0 @@ --- update_data - diff --git a/models/world_wb_mides/world_wb_mides__raw_empenho_ce.sql b/models/world_wb_mides/world_wb_mides__raw_empenho_ce.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_empenho_df.sql b/models/world_wb_mides/world_wb_mides__raw_empenho_df.sql deleted file mode 100644 index 53d94721e8..0000000000 --- a/models/world_wb_mides/world_wb_mides__raw_empenho_df.sql +++ /dev/null @@ -1,2 +0,0 @@ --- update_data - diff --git a/models/world_wb_mides/world_wb_mides__raw_empenho_mg.sql b/models/world_wb_mides/world_wb_mides__raw_empenho_mg.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_empenho_pb.sql b/models/world_wb_mides/world_wb_mides__raw_empenho_pb.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_empenho_pe.sql b/models/world_wb_mides/world_wb_mides__raw_empenho_pe.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_empenho_pr.sql b/models/world_wb_mides/world_wb_mides__raw_empenho_pr.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_empenho_rj.sql b/models/world_wb_mides/world_wb_mides__raw_empenho_rj.sql deleted file mode 100644 index 53d94721e8..0000000000 --- a/models/world_wb_mides/world_wb_mides__raw_empenho_rj.sql +++ /dev/null @@ -1,2 +0,0 @@ --- update_data - diff --git a/models/world_wb_mides/world_wb_mides__raw_empenho_sc.sql b/models/world_wb_mides/world_wb_mides__raw_empenho_sc.sql deleted file mode 100644 index 53d94721e8..0000000000 --- a/models/world_wb_mides/world_wb_mides__raw_empenho_sc.sql +++ /dev/null @@ -1,2 +0,0 @@ --- update_data - diff --git a/models/world_wb_mides/world_wb_mides__raw_estorno_pb.sql b/models/world_wb_mides/world_wb_mides__raw_estorno_pb.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_liquidacao_ce.sql b/models/world_wb_mides/world_wb_mides__raw_liquidacao_ce.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_liquidacao_df.sql b/models/world_wb_mides/world_wb_mides__raw_liquidacao_df.sql deleted file mode 100644 index 53d94721e8..0000000000 --- a/models/world_wb_mides/world_wb_mides__raw_liquidacao_df.sql +++ /dev/null @@ -1,2 +0,0 @@ --- update_data - diff --git a/models/world_wb_mides/world_wb_mides__raw_liquidacao_mg.sql b/models/world_wb_mides/world_wb_mides__raw_liquidacao_mg.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_liquidacao_pb.sql b/models/world_wb_mides/world_wb_mides__raw_liquidacao_pb.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_liquidacao_pe.sql b/models/world_wb_mides/world_wb_mides__raw_liquidacao_pe.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_liquidacao_pr.sql b/models/world_wb_mides/world_wb_mides__raw_liquidacao_pr.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_liquidacao_rj.sql b/models/world_wb_mides/world_wb_mides__raw_liquidacao_rj.sql deleted file mode 100644 index 53d94721e8..0000000000 --- a/models/world_wb_mides/world_wb_mides__raw_liquidacao_rj.sql +++ /dev/null @@ -1,2 +0,0 @@ --- update_data - diff --git a/models/world_wb_mides/world_wb_mides__raw_pagamento_ce.sql b/models/world_wb_mides/world_wb_mides__raw_pagamento_ce.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_pagamento_df.sql b/models/world_wb_mides/world_wb_mides__raw_pagamento_df.sql deleted file mode 100644 index 53d94721e8..0000000000 --- a/models/world_wb_mides/world_wb_mides__raw_pagamento_df.sql +++ /dev/null @@ -1,2 +0,0 @@ --- update_data - diff --git a/models/world_wb_mides/world_wb_mides__raw_pagamento_mg.sql b/models/world_wb_mides/world_wb_mides__raw_pagamento_mg.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_pagamento_pb.sql b/models/world_wb_mides/world_wb_mides__raw_pagamento_pb.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_pagamento_pe.sql b/models/world_wb_mides/world_wb_mides__raw_pagamento_pe.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_pagamento_pr.sql b/models/world_wb_mides/world_wb_mides__raw_pagamento_pr.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_pagamento_rj.sql b/models/world_wb_mides/world_wb_mides__raw_pagamento_rj.sql deleted file mode 100644 index 53d94721e8..0000000000 --- a/models/world_wb_mides/world_wb_mides__raw_pagamento_rj.sql +++ /dev/null @@ -1,2 +0,0 @@ --- update_data - diff --git a/models/world_wb_mides/world_wb_mides__raw_resumo_pe.sql b/models/world_wb_mides/world_wb_mides__raw_resumo_pe.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_rsp_mg.sql b/models/world_wb_mides/world_wb_mides__raw_rsp_mg.sql deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/models/world_wb_mides/world_wb_mides__raw_rsp_sp.sql b/models/world_wb_mides/world_wb_mides__raw_rsp_sp.sql deleted file mode 100644 index e69de29bb2..0000000000 From aca26c714e27171e3864a3c39345b5018287cac3 Mon Sep 17 00:00:00 2001 From: Ricardo Dahis Date: Mon, 24 Aug 2026 17:08:52 +1000 Subject: [PATCH 2/6] refactor(world_wb_mides): one model per state behind each published table empenho.sql (101 KB), pagamento.sql (72 KB) and liquidacao.sql (50 KB) were single chains of per-state CTEs -- 57, 49 and 36 of them -- unioned at the end. Adding a state meant editing a 2,000-line file, and one state's bad column broke the table for every state. Each is now a union over ten per-state models under models/world_wb_mides/states/, and the parents are ~60 lines each. The CTE dependency graph turned out to be cleanly separable: every state's transitive closure is disjoint from every other's, with no shared helper CTEs, so no logic had to be rewritten to split them. Materialisation is ephemeral, set in dbt_project.yml. The state models inline as CTEs exactly as before, so the compiled SQL, the published tables and the storage footprint are all unchanged. Switching to `table` in a non-published schema is a two-line change there, and is what the per-state pipelines will want so a state can be rebuilt without touching the rest -- deliberately left for that work, since it roughly doubles the ~230 GB footprint. The state models deliberately do NOT project the canonical column list. Column names come from the first term of the parent union, as they always have, and several states alias columns differently inside their own CTEs -- empenho_ce calls one `modalidade_despesa` where the table calls it `elemento_despesa`, and liquidacao_df calls one `month` where the table calls it `mes`. The union has always resolved that positionally. Projecting the canonical list per state looked tidier but changed behaviour, and BigQuery rejected it; the parent keeps the job. Those aliases are misleading and worth renaming, but not inside a refactor that is meant to change nothing. Also drops the `frequencia` CTE from liquidacao and pagamento: defined in both, referenced by neither (the later `count(1) as frequencia` is a column alias, not a reference), so it fell outside every state's closure. Verified three ways: - static: every CTE body token-identical to its pre-split source, each CTE in exactly one state, union terms an identical multiset, per file. - BigQuery dry run: all 30 state models parse and resolve; all three parents return the same output schema and column names as the pre-split compiled SQL. - end-to-end: world_wb_mides__empenho_df built against dev matches production exactly -- 870,042 rows, 870,042 distinct empenhos, sum(valor_final) 309,418,235,890.57 on both sides. --- dbt_project.yml | 6 + .../states/world_wb_mides__empenho_ce.sql | 133 + .../states/world_wb_mides__empenho_df.sql | 104 + .../states/world_wb_mides__empenho_mg.sql | 95 + .../states/world_wb_mides__empenho_pb.sql | 246 ++ .../states/world_wb_mides__empenho_pe.sql | 309 +++ .../states/world_wb_mides__empenho_pr.sql | 52 + .../states/world_wb_mides__empenho_rj.sql | 424 ++++ .../states/world_wb_mides__empenho_rs.sql | 261 ++ .../states/world_wb_mides__empenho_sc.sql | 115 + .../states/world_wb_mides__empenho_sp.sql | 363 +++ .../states/world_wb_mides__liquidacao_ce.sql | 51 + .../states/world_wb_mides__liquidacao_df.sql | 62 + .../states/world_wb_mides__liquidacao_mg.sql | 87 + .../states/world_wb_mides__liquidacao_pb.sql | 55 + .../states/world_wb_mides__liquidacao_pe.sql | 37 + .../states/world_wb_mides__liquidacao_pr.sql | 43 + .../states/world_wb_mides__liquidacao_rj.sql | 256 ++ .../states/world_wb_mides__liquidacao_rs.sql | 165 ++ .../states/world_wb_mides__liquidacao_sc.sql | 84 + .../states/world_wb_mides__liquidacao_sp.sql | 269 ++ .../states/world_wb_mides__pagamento_ce.sql | 145 ++ .../states/world_wb_mides__pagamento_df.sql | 107 + .../states/world_wb_mides__pagamento_mg.sql | 118 + .../states/world_wb_mides__pagamento_pb.sql | 129 + .../states/world_wb_mides__pagamento_pe.sql | 52 + .../states/world_wb_mides__pagamento_pr.sql | 53 + .../states/world_wb_mides__pagamento_rj.sql | 297 +++ .../states/world_wb_mides__pagamento_rs.sql | 250 ++ .../states/world_wb_mides__pagamento_sc.sql | 93 + .../states/world_wb_mides__pagamento_sp.sql | 367 +++ .../world_wb_mides__empenho.sql | 2206 +---------------- .../world_wb_mides__liquidacao.sql | 1111 +-------- .../world_wb_mides__pagamento.sql | 1648 +----------- 34 files changed, 4885 insertions(+), 4908 deletions(-) create mode 100644 models/world_wb_mides/states/world_wb_mides__empenho_ce.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__empenho_df.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__empenho_mg.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__empenho_pb.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__empenho_pe.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__empenho_pr.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__empenho_rj.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__empenho_rs.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__empenho_sc.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__empenho_sp.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__liquidacao_ce.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__liquidacao_df.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__liquidacao_mg.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__liquidacao_pb.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__liquidacao_pe.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__liquidacao_pr.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__liquidacao_rj.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__liquidacao_rs.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__liquidacao_sc.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__liquidacao_sp.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__pagamento_ce.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__pagamento_df.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__pagamento_mg.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__pagamento_pb.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__pagamento_pe.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__pagamento_pr.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__pagamento_rj.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__pagamento_rs.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__pagamento_sc.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__pagamento_sp.sql diff --git a/dbt_project.yml b/dbt_project.yml index cae8a87fab..cadb2ab3a1 100644 --- a/dbt_project.yml +++ b/dbt_project.yml @@ -583,6 +583,12 @@ models: world_wb_mides: +materialized: table +schema: world_wb_mides + # One model per state feeds each published table. Ephemeral keeps them as + # inlined CTEs, so the compiled SQL and the published tables are unchanged + # by the split. Switch to `table` (plus a non-published +schema) when the + # per-state pipelines need to rebuild one state without touching the rest. + states: + +materialized: ephemeral world_wwf_hydrosheds: +materialized: table +schema: world_wwf_hydrosheds diff --git a/models/world_wb_mides/states/world_wb_mides__empenho_ce.sql b/models/world_wb_mides/states/world_wb_mides__empenho_ce.sql new file mode 100644 index 0000000000..04fc7b5e96 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__empenho_ce.sql @@ -0,0 +1,133 @@ +-- Ceará (CE) contribution to world_wb_mides.empenho. +-- Split out of the monolithic empenho model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + empenhado_ce as ( + select + (safe_cast(extract(year from date(data_emissao_empenho)) as int64)) as ano, + (safe_cast(extract(month from date(data_emissao_empenho)) as int64)) as mes, + safe_cast( + extract(date from timestamp(data_emissao_empenho)) as date + ) as data, + 'CE' as sigla_uf, + safe_cast(geoibgeid as string) as id_municipio, + safe_cast(codigo_orgao as string) as orgao, + safe_cast(trim(codigo_unidade) as string) as id_unidade_gestora, + safe_cast(null as string) as id_licitacao_bd, + safe_cast(numero_licitacao as string) as id_licitacao, + case + when tipo_processo_licitatorio = 'N' + then '98' + when tipo_processo_licitatorio = 'R' + then '2' + when tipo_processo_licitatorio = 'D' + then '8' + when tipo_processo_licitatorio = 'I' + then '10' + when tipo_processo_licitatorio = 'R' + then '29' + end as modalidade_licitacao, + safe_cast( + concat( + numero_empenho, + ' ', + trim(codigo_orgao), + ' ', + trim(codigo_unidade), + ' ', + geoibgeid, + ' ', + (substring(data_emissao_empenho, 6, 2)), + ' ', + (substring(data_emissao_empenho, 3, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(numero_empenho as string) as numero, + safe_cast(lower(descricao_empenho) as string) as descricao, + safe_cast(modalidade_empenho as string) as modalidade, + safe_cast(safe_cast(codigo_funcao as int64) as string) as funcao, + safe_cast(safe_cast(codigo_subfuncao as int64) as string) as subfuncao, + safe_cast(safe_cast(codigo_programa as int64) as string) as programa, + safe_cast(safe_cast(codigo_projeto_atividade as int64) as string) as acao, + safe_cast( + safe_cast(codigo_elemento_despesa as int64) as string + ) as modalidade_despesa, + round(safe_cast(valor_empenhado as float64), 2) as valor_inicial, + from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_ce") }} e + + ), + anulacao_ce as ( + select + safe_cast( + concat( + numero_empenho, + ' ', + trim(codigo_orgao), + ' ', + trim(codigo_unidade), + ' ', + geoibgeid, + ' ', + (substring(data_emissao_empenho, 6, 2)), + ' ', + (substring(data_emissao_empenho, 3, 2)) + ) as string + ) as id_empenho_bd, + round(sum(safe_cast(valor_anulacao as float64)), 2) as valor_anulacao + from {{ set_datalake_project("world_wb_mides_staging.raw_anulacao_ce") }} + group by 1 + + ), + frequencia_ce as ( + select id_empenho_bd, count(id_empenho_bd) as frequencia_id + from empenhado_ce + group by 1 + order by 2 desc + + ), + empenho_ce as ( + select + e.ano, + e.mes, + e.data, + e.sigla_uf, + e.id_municipio, + e.orgao, + e.id_unidade_gestora, + e.id_licitacao_bd, + e.id_licitacao, + e.modalidade_licitacao, + ( + case + when frequencia_id > 1 + then (safe_cast(null as string)) + else e.id_empenho_bd + end + ) as id_empenho_bd, + e.id_empenho, + e.numero, + e.descricao, + e.modalidade, + e.funcao, + e.subfuncao, + e.programa, + e.acao, + e.modalidade_despesa, + round(e.valor_inicial, 2), + round(safe_cast(0 as float64), 2) as valor_reforco, + round(a.valor_anulacao, 2), + round(safe_cast(0 as float64), 2) as valor_ajuste, + round( + ifnull(e.valor_inicial, 0) - ifnull(a.valor_anulacao, 0), 2 + ) as valor_final + from empenhado_ce e + left join frequencia_ce f on e.id_empenho_bd = f.id_empenho_bd + full outer join anulacao_ce a on a.id_empenho_bd = e.id_empenho_bd + + ) +select * +from empenho_ce diff --git a/models/world_wb_mides/states/world_wb_mides__empenho_df.sql b/models/world_wb_mides/states/world_wb_mides__empenho_df.sql new file mode 100644 index 0000000000..0cff28bae3 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__empenho_df.sql @@ -0,0 +1,104 @@ +-- Distrito Federal (DF) contribution to world_wb_mides.empenho. +-- Split out of the monolithic empenho model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + empenho_df as ( + select + (safe_cast(exercicio as int64)) as ano, + (safe_cast(extract(month from date(lancamento)) as int64)) as mes, + safe_cast(lancamento as date) as data, + 'DF' as sigla_uf, + '5300108' as id_municipio, + safe_cast(codigo_ug as string) as orgao, + safe_cast(codigo_gestao as string) as id_unidade_gestora, + safe_cast(null as string) as id_licitacao_bd, + safe_cast(null as string) as id_licitacao, + case + when codigo_licitacao = '1' + then '11' + when codigo_licitacao = '2' + then '1' + when codigo_licitacao = '3' + then '2' + when codigo_licitacao = '4' + then '3' + when codigo_licitacao = '5' + then '8' + when codigo_licitacao = '6' + then '10' + when codigo_licitacao = '7' + then '99' + when codigo_licitacao = '8' + then '32' + when codigo_licitacao = '9' + then '4' + when codigo_licitacao = '10' + then '32' + when codigo_licitacao = '11' + then '31' + when codigo_licitacao = '12' + then '' + when codigo_licitacao = '13' + then '5' + when codigo_licitacao = '14' + then '6' + when codigo_licitacao = '15' + then '5' + when codigo_licitacao = '16' + then '5' + when codigo_licitacao = '17' + then '6' + when codigo_licitacao = '18' + then '3' + when codigo_licitacao = '19' + then '32' + when codigo_licitacao = '20' + then '31' + when codigo_licitacao = '21' + then '31' + when codigo_licitacao = '22' + then '32' + when codigo_licitacao = '23' + then '12' + when codigo_licitacao = '25' + then '98' + when codigo_licitacao = 'INEXIGÍVEL' + then '10' + end as modalidade_licitacao, + safe_cast( + concat( + right(nota_empenho, length(nota_empenho) - 6), + ' ', + codigo_ug, + ' ', + codigo_gestao, + ' ', + '5300108', + ' ', + (right(exercicio, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(nota_empenho as string) as numero, + safe_cast(descricao as string) as descricao, + safe_cast(left(modalidade_empenho, 1) as string) as modalidade, + safe_cast(cast(codigo_funcao as int64) as string) as funcao, + safe_cast(codigo_subfuncao as string) as subfuncao, + safe_cast(codigo_programa as string) as programa, + safe_cast(codigo_acao as string) as acao, + safe_cast(codigo_natureza as string) as elemento_despesa, + round(safe_cast(0 as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_reforco, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round( + safe_cast(replace (valor_final, ',', '.') as float64), 2 + ) as valor_final + from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_df") }} + + ) +select * +from empenho_df diff --git a/models/world_wb_mides/states/world_wb_mides__empenho_mg.sql b/models/world_wb_mides/states/world_wb_mides__empenho_mg.sql new file mode 100644 index 0000000000..28cd7ae372 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__empenho_mg.sql @@ -0,0 +1,95 @@ +-- Minas Gerais (MG) contribution to world_wb_mides.empenho. +-- Split out of the monolithic empenho model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + empenhado_mg as ( + select + safe_cast(ano as int64) as ano, + safe_cast(mes as int64) as mes, + safe_cast(data as date) as data, + 'MG' as sigla_uf, + safe_cast(id_municipio as string) as id_municipio, + safe_cast(trim(orgao) as string) as orgao, + safe_cast(id_unidade_gestora as string) as id_unidade_gestora, + safe_cast(null as string) as id_licitacao_bd, + safe_cast(id_licitacao as string) as id_licitacao, + safe_cast(null as string) as modalidade_licitacao, + safe_cast( + concat( + id_empenho, ' ', orgao, ' ', id_municipio, ' ', (right(ano, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(id_empenho as string) as id_empenho, + safe_cast(numero_empenho as string) as numero, + safe_cast(lower(descricao) as string) as descricao, + safe_cast(substring(dsc_modalidade, 5, 1) as string) as modalidade, + safe_cast(cast(left(dsc_funcao, 2) as int64) as string) as funcao, + safe_cast(cast(left(dsc_subfuncao, 3) as int64) as string) as subfuncao, + safe_cast(cast(left(dsc_programa, 4) as int64) as string) as programa, + safe_cast(cast(left(dsc_acao, 4) as int64) as string) as acao, + safe_cast( + replace(left(elemento_despesa, 12), '.', '') as string + ) as elemento_despesa, + round(safe_cast(valor_empenho_original as float64), 2) as valor_inicial, + round( + safe_cast(ifnull(safe_cast(valor_reforco as float64), 0) as float64), 2 + ) as valor_reforco, + round( + safe_cast(ifnull(safe_cast(valor_anulacao as float64), 0) as float64), 2 + ) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round( + safe_cast(valor_empenho_original as float64) + + safe_cast(ifnull(safe_cast(valor_reforco as float64), 0) as float64) + - safe_cast(ifnull(safe_cast(valor_anulacao as float64), 0) as float64), + 2 + ) as valor_final + from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_mg") }} + + ), + dlic as ( + select + id_empenho_bd, + case when (count(distinct id_licitacao)) > 1 then 1 else 0 end as dlic + from empenhado_mg + group by 1 + + ), + empenho_mg as ( + select distinct + e.ano, + e.mes, + e.data, + e.sigla_uf, + e.id_municipio, + e.orgao, + e.id_unidade_gestora, + e.id_licitacao_bd, + case + when dlic = 1 then (safe_cast(null as string)) else e.id_licitacao + end as id_licitacao, + e.modalidade_licitacao, + e.id_empenho_bd, + e.id_empenho, + e.numero, + e.descricao, + e.modalidade, + e.funcao, + e.subfuncao, + e.programa, + e.acao, + e.elemento_despesa, + e.valor_inicial, + e.valor_reforco, + e.valor_anulacao, + e.valor_ajuste, + e.valor_final + from empenhado_mg e + left join dlic l on l.id_empenho_bd = e.id_empenho_bd + + ) +select * +from empenho_mg diff --git a/models/world_wb_mides/states/world_wb_mides__empenho_pb.sql b/models/world_wb_mides/states/world_wb_mides__empenho_pb.sql new file mode 100644 index 0000000000..18578354bb --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__empenho_pb.sql @@ -0,0 +1,246 @@ +-- Paraíba (PB) contribution to world_wb_mides.empenho. +-- Split out of the monolithic empenho model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + empenhado_pb as ( + select + safe_cast(dt_ano as int64) as ano, + safe_cast(substring(trim(dt_empenho), -7, 2) as int64) as mes, + safe_cast( + concat( + substring(trim(dt_empenho), -4), + '-', + substring(trim(dt_empenho), -7, 2), + '-', + substring(trim(dt_empenho), 1, 2) + ) as date + ) as data, + 'PB' as sigla_uf, + safe_cast(m.id_municipio as string) as id_municipio, + safe_cast(e.cd_ugestora as string) as orgao, + safe_cast(null as string) as id_unidade_gestora, + safe_cast(null as string) as id_licitacao_bd, + safe_cast(null as string) as id_licitacao, + safe_cast(null as string) as modalidade_licitacao, + safe_cast( + concat( + nu_empenho, + ' ', + e.cd_ugestora, + ' ', + m.id_municipio, + ' ', + (right(dt_ano, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(nu_empenho as string) as numero, + safe_cast(lower(de_historico) as string) as descricao, + safe_cast(null as string) as modalidade, + safe_cast(safe_cast(funcao as int64) as string) as funcao, + safe_cast(safe_cast(subfuncao as int64) as string) as subfuncao, + safe_cast(de_programa as string) as programa, -- substituir por código + safe_cast(de_acao as string) as acao, -- substituir por código + concat( + case + when de_cateconomica = 'Despesa Corrente' + then '3' + when de_cateconomica = 'Despesa de Capital' + then '4' + when de_cateconomica = 'Reserva de Contingência' + then '9' + end, + case + when de_natdespesa = 'Pessoal e Encargos Sociais' + then '1' + when de_natdespesa = 'Juros e Encargos da Dívida' + then '2' + when de_natdespesa = 'Outras Despesas Correntes' + then '3' + when de_natdespesa = 'Investimentos' + then '4' + when de_natdespesa = 'Inversões Financeiras' + then '5' + when de_natdespesa = 'Amortização da Dívida' + then '6' + when de_natdespesa = 'Reserva de Contingência' + then '9' + end, + case + when de_modalidade = 'Transferências à União' + then '20' + when + de_modalidade + = 'Transferências a Instituições Privadas com Fins Lucrativos' + then '30' + when + de_modalidade + = 'Execução Orçamentária Delegada a Estados e ao Distrito Federal' + then '32' + when + de_modalidade = 'Aplicação Direta §§ 1º e 2º do Art. 24 LC 1412' + then '35' + when de_modalidade = 'Aplicação Direta Art. 25 LC 141' + then '36' + when de_modalidade = 'Transferências a Municípios' + then '40' + when de_modalidade = 'Transferências a Municípios – Fundo a Fundo' + then '41' + when + de_modalidade + = 'Transferências a Instituições Privadas sem Fins Lucrativos' + then '50' + when + de_modalidade + = 'Transferências a Instituições Privadas com Fins Lucrativos' + then '60' + when + de_modalidade + = 'Transferências a Instituições Multigovernamentais' + then '70' + when + de_modalidade + = 'Transf. a Consórc Púb. C.Rateio §§ 1º e 2º Art. 24 LC141' + then '71' + when + de_modalidade + = 'Execução Orçamentária Delegada a Consórcios Públicos' + then '72' + when de_modalidade = 'Transferências a Consórcios Públicos' + then '73' + when + de_modalidade = 'Transf. a Consórc Púb. C.Rateio Art. 25 LC 141' + then '74' + when de_modalidade = 'Transferências ao Exterior' + then '80' + when de_modalidade = 'Aplicações Diretas' + then '90' + when + de_modalidade + = 'Ap. Direta Decor. de Op. entre Órg., Fundos e Ent. Integ. dos Orçamentos Fiscal e da Seguridade Social' + then '91' + when + de_modalidade + = ' Aplicação Direta Decor. de Oper. de Órgãos, Fundos e Entid. Integr. dos Orç. Fiscal e da Seguri. Social com Cons. Públ. do qual o Ente Participe' + then '93' + when + de_modalidade + = ' Aplicação Direta Decor. de Oper. de Órgãos, Fundos e Entid. Integr. dos Orç. Fiscal e da Seguri. Social com Cons. Públ. do qual o Ente Não Participe' + then '94' + else null + end, + cd_elemento + ) as elemento_despesa, + safe_cast(vl_empenho as float64) as valor_inicial + from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_pb") }} e + left join + {{ set_datalake_project("world_wb_mides_staging.aux_municipio_pb") }} m + on e.cd_ugestora = safe_cast(m.id_unidade_gestora as string) + left join + {{ set_datalake_project("world_wb_mides_staging.aux_funcao") }} f + on e.de_funcao = f.nome_funcao + left join + {{ set_datalake_project("world_wb_mides_staging.aux_subfuncao") }} sf + on e.de_subfuncao = sf.nome_subfuncao + + ), + anulacao_pb as ( + select + safe_cast( + concat( + nu_empenho, + ' ', + a.cd_ugestora, + ' ', + m.id_municipio, + ' ', + (right(dt_ano, 2)) + ) as string + ) as id_empenho_bd, + sum(safe_cast(vl_estorno as float64)) as valor_anulacao + from {{ set_datalake_project("world_wb_mides_staging.raw_estorno_pb") }} a + left join + {{ set_datalake_project("world_wb_mides_staging.aux_municipio_pb") }} m + on a.cd_ugestora = safe_cast(m.id_unidade_gestora as string) + group by 1 + + ), + frequencia_pb as ( + select id_empenho_bd, count(id_empenho_bd) as frequencia_id + from empenhado_pb + group by 1 + + ), + empenho_completo as ( + select + e.ano, + e.mes, + e.data, + e.sigla_uf, + e.id_municipio, + e.orgao, + e.id_unidade_gestora, + e.id_licitacao_bd, + e.id_licitacao, + e.modalidade_licitacao, + e.id_empenho_bd, + e.id_empenho, + e.numero, + e.descricao, + e.modalidade, + e.funcao, + e.subfuncao, + e.programa, + e.acao, + e.elemento_despesa, + frequencia_id, + round(sum(e.valor_inicial), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_reforco, + round(sum(a.valor_anulacao / frequencia_id), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + from empenhado_pb e + full outer join anulacao_pb a on a.id_empenho_bd = e.id_empenho_bd + left join frequencia_pb f on f.id_empenho_bd = e.id_empenho_bd + group by + 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21 + + ), + empenho_pb as ( + select + e.ano, + e.mes, + e.data, + e.sigla_uf, + e.id_municipio, + e.orgao, + e.id_unidade_gestora, + e.id_licitacao_bd, + e.id_licitacao, + e.modalidade_licitacao, + case + when (frequencia_id > 1) + then (safe_cast(null as string)) + else e.id_empenho_bd + end as id_empenho_bd, + e.id_empenho, + e.numero, + e.descricao, + e.modalidade, + e.funcao, + e.subfuncao, + e.programa, + e.acao, + e.elemento_despesa, + e.valor_inicial, + e.valor_reforco, + e.valor_anulacao, + e.valor_ajuste, + round(e.valor_inicial - ifnull(valor_anulacao, 0), 2) as valor_final + from empenho_completo e + + ) +select * +from empenho_pb diff --git a/models/world_wb_mides/states/world_wb_mides__empenho_pe.sql b/models/world_wb_mides/states/world_wb_mides__empenho_pe.sql new file mode 100644 index 0000000000..9fa02b071a --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__empenho_pe.sql @@ -0,0 +1,309 @@ +-- Pernambuco (PE) contribution to world_wb_mides.empenho. +-- Split out of the monolithic empenho model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + empenho_pe as ( + select + safe_cast(e.anoreferencia as int64) as ano, + (safe_cast(extract(month from date(dataempenho)) as int64)) as mes, + safe_cast(extract(date from timestamp(dataempenho)) as date) as data, + 'PE' as sigla_uf, + safe_cast(codigoibge as string) as id_municipio, + safe_cast(null as string) orgao, + safe_cast(id_unidadegestora as string) as id_unidade_gestora, + safe_cast(null as string) id_licitacao_bd, + safe_cast(null as string) id_licitacao, + safe_cast(null as string) modalidade_licitacao, + safe_cast(null as string) as id_empenho_bd, + safe_cast(trim(id_empenho) as string) as id_empenho, + safe_cast(e.numeroempenho as string) as numero, + safe_cast(lower(historico) as string) as descricao, + safe_cast(left(tipo_empenho, 1) as string) as modalidade, + safe_cast(safe_cast(fun.funcao as int64) as string) as funcao, + safe_cast(safe_cast(sub.subfuncao as int64) as string) as subfuncao, + safe_cast(programa as string) as programa, + safe_cast(codigo_tipo_acao as string) as acao, + concat( + case + when categoria = 'Despesa Corrente' + then '3' + when categoria = 'Despesa de Capital' + then '4' + end, + case + when natureza = 'Pessoal e Encargos Sociais' + then '1' + when natureza = 'Juros e Encargos da Dívida' + then '2' + when natureza = 'Outras Despesas Correntes' + then '3' + when natureza = 'Investimentos' + then '4' + when natureza = 'Inversões Financeiras' + then '5' + when natureza = 'Amortização da Dívida' + then '6' + when natureza = 'Reserva de Contingência' + then '9' + end, + case + when modalidade = 'Transferências à União' + then '20' + when + modalidade + = 'Transferências a Instituições Privadas com Fins Lucrativos' + then '30' + when + modalidade + = 'Execução Orçamentária Delegada a Estados e ao Distrito Federal' + then '32' + when + modalidade + = 'Aplicação Direta à conta de recursos de que tratam os §§ 1o e 2o do art. 24 da Lei Complementar no 141, de 2012' + then '35' + when + modalidade + = 'Aplicação Direta à conta de recursos de que trata o art. 25 da Lei Complementar no 141, de 2012' + then '36' + when modalidade = 'Transferências a Municípios' + then '40' + when modalidade = 'Transferências a Municípios – Fundo a Fundo' + then '41' + when + modalidade + = 'Transferências a Instituições Privadas sem Fins Lucrativos' + then '50' + when + modalidade + = 'Transferências a Instituições Privadas com Fins Lucrativos' + then '60' + when + modalidade = 'Transferências a Instituições Multigovernamentais' + then '70' + when + modalidade + = 'Transferências a Consórcios Públicos mediante contrato de rateio à conta de recursos de que tratam os §§ 1o e 2o do art. 24 da Lei Complementar no 141, de 2012' + then '71' + when + modalidade + = 'Execução Orçamentária Delegada a Consórcios Públicos' + then '72' + when modalidade = 'Transferências a Consórcios Públicos' + then '73' + when modalidade = 'Transferências ao Exterior' + then '80' + when modalidade = 'Aplicações Diretas' + then '90' + when + modalidade + = 'Ap. Direta Decor. de Op. entre Órg., Fundos e Ent. Integ. dos Orçamentos Fiscal e da Seguridade Social' + then '91' + when + modalidade + = ' Aplicação Direta Decor. de Oper. de Órgãos, Fundos e Entid. Integr. dos Orç. Fiscal e da Seguri. Social com Cons. Públ. do qual o Ente Participe' + then '93' + when + modalidade + = ' Aplicação Direta Decor. de Oper. de Órgãos, Fundos e Entid. Integr. dos Orç. Fiscal e da Seguri. Social com Cons. Públ. do qual o Ente Não Participe' + then '94' + else null + end, + case + when elementodespesa = 'Pensões do RPPS e do militar' + then '03' + when elementodespesa = 'Contratação por Tempo Determinado' + then '04' + when elementodespesa = 'Outros Benefícios Previdenciários do RPPS' + then '05' + when + elementodespesa + = 'Outros Benefícios Previdenciários do servidor ou do militar' + then '05' + when elementodespesa = 'Beneficio Mensal ao Deficiente e ao Idoso' + then '06' + when + elementodespesa + = 'Contribuição a Entidades Fechadas de Previdência' + then '07' + when elementodespesa = 'Outros Benefícios Assistenciais' + then '08' + when + elementodespesa + = 'Outros Benefícios Assistenciais do servidor e do militar' + then '08' + when elementodespesa = 'Salário Família' + then '09' + when elementodespesa = 'Seguro Desemprego e Abono Salarial' + then '10' + when + elementodespesa + = 'Vencimentos e Vantagens Fixas - Pessoal Civil' + then '11' + when + elementodespesa + = 'Vencimentos e Vantagens Fixas - Pessoal Militar' + then '12' + when elementodespesa = 'Obrigações Patronais' + then '13' + when + elementodespesa + = 'Aporte para Cobertura do Déficit Atuarial do RPPS' + then '13' + when elementodespesa = 'Diárias - Civil' + then '14' + when elementodespesa = 'Outras Despesas Variáveis - Pessoal Civil' + then '16' + when elementodespesa = 'Auxílio Financeiro a Estudantes' + then '18' + when elementodespesa = 'Auxílio Fardamento' + then '19' + when elementodespesa = 'Auxílio Financeiro a Pesquisadores' + then '20' + when elementodespesa = 'Outros Encargos sobre a Dívida por Contrato' + then '22' + when + elementodespesa + = 'Juros, Deságios e Descontos da Dívida Mobiliária' + then '23' + when elementodespesa = 'Outros Encargos sobre a Dívida Mobiliária' + then '24' + when + elementodespesa + = 'Encargos sobre Operações de Crédito por Antecipação da Receita' + then '25' + when + elementodespesa + = 'Encargos pela Honra de Avais, Garantias, Seguros e Similares' + then '27' + when elementodespesa = 'Remuneração de Cotas de Fundos Autárquicos' + then '28' + when elementodespesa = 'Material de Consumo' + then '30' + when + elementodespesa + = 'Premiações Culturais, Artísticas, Científicas, Desportivas e Outras' + then '31' + when + elementodespesa + = 'Material, Bem ou Serviço para Distribuição Gratuita' + then '32' + when elementodespesa = 'Passagens e Despesas de Locomoção' + then '33' + when + elementodespesa + = 'Outras Despesas de Pessoal decorrentes de Contratos de Terceirização' + then '34' + when elementodespesa = 'Serviços de Consultoria' + then '35' + when elementodespesa = 'Locação de Mão-de-Obra' + then '37' + when + elementodespesa + = 'Outros Serviços de Terceiros ? Pessoa Jurídica' + then '39' + when + elementodespesa + = 'Serviços de Tecnologia da Informação e Comunicação - Pessoa Jurídica' + then '40' + when + elementodespesa + = 'Serviços de Tecnologia da Informação e Comunicação ? Pessoa Jurídica' + then '40' + when elementodespesa = 'Contribuições' + then '41' + when elementodespesa = 'Auxílios' + then '42' + when elementodespesa = 'Obrigações Tributárias e Contributivas' + then '47' + when elementodespesa = 'Auxílio-Transporte' + then '49' + when elementodespesa = 'Obras e Instalações' + then '51' + when elementodespesa = 'Equipamentos e Material Permanente' + then '52' + when elementodespesa = 'Aposentadorias do RGPS ? Área Urbana' + then '54' + when elementodespesa = 'Pensões, exclusiva do RGPS' + then '56' + when elementodespesa = 'Outros Benefícios do RGPS ? Área Urbana' + then '58' + when elementodespesa = 'Pensões Especiais' + then '59' + when elementodespesa = 'Aquisição de Imóveis' + then '61' + when + elementodespesa + = 'Constituição ou Aumento de Capital de Empresas' + then '65' + when elementodespesa = 'Concessão de Empréstimos e Financiamentos' + then '66' + when elementodespesa = 'Depósitos Compulsórios' + then '67' + when + elementodespesa + = 'Rateio pela Participação em Consórcio Público' + then '70' + when elementodespesa = 'Principal da Dívida Contratual Resgatado' + then '71' + when elementodespesa = 'Principal da Dívida Mobiliária Resgatado' + then '72' + when + elementodespesa + = 'Correção Monetária ou Cambial da Dívida Contratual Resgatada' + then '73' + when + elementodespesa + = 'Principal Corrigido da Dívida Contratual Refinanciado' + then '77' + when + elementodespesa + = 'Distribuição Constitucional ou Legal de Receitas' + then '81' + when elementodespesa = 'Sentenças Judiciais' + then '91' + when elementodespesa = 'Despesas de Exercícios Anteriores' + then '92' + when elementodespesa = 'Indenizações e Restituições' + then '93' + when + elementodespesa + = 'Indenização pela Execução de Trabalhos de Campo' + then '95' + when + elementodespesa + = 'Ressarcimento de Despesas de Pessoal Requisitado' + then '96' + else null + end + ) as elemento_despesa, + round(safe_cast(0 as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_reforco, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(valorempenhado as float64), 2) as valor_final + from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_pe") }} e + left join + {{ set_datalake_project("world_wb_mides_staging.aux_municipio_pe") }} m + on e.nomeunidadegestora = m.nomeunidadegestora + left join + {{ set_datalake_project("world_wb_mides_staging.aux_funcao") }} fun + on upper( + trim( + replace( + replace(e.funcao, 'Encargos Especias', 'Encargos Especiais'), + 'Assistêncial Social', + 'Assistência Social' + ) + ) + ) + = upper(nome_funcao) + left join + {{ set_datalake_project("world_wb_mides_staging.aux_subfuncao") }} sub + on upper(trim(e.subfuncao)) = upper(nome_subfuncao) + + ) +select * +from empenho_pe diff --git a/models/world_wb_mides/states/world_wb_mides__empenho_pr.sql b/models/world_wb_mides/states/world_wb_mides__empenho_pr.sql new file mode 100644 index 0000000000..7602592e61 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__empenho_pr.sql @@ -0,0 +1,52 @@ +-- Paraná (PR) contribution to world_wb_mides.empenho. +-- Split out of the monolithic empenho model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + empenho_pr as ( + select + safe_cast(nranoempenho as int64) as ano, + (safe_cast(extract(month from date(dtempenho)) as int64)) as mes, + safe_cast(extract(date from timestamp(dtempenho)) as date) as data, + 'PR' as sigla_uf, + safe_cast(m.id_municipio as string) as id_municipio, + safe_cast(trim(cdorgao, '0') as string) as orgao, + safe_cast(cdunidade as string) as id_unidade_gestora, + safe_cast(null as string) as id_licitacao_bd, + safe_cast(null as string) as id_licitacao, + safe_cast(null as string) as modalidade_licitacao, + safe_cast( + concat(idempenho, ' ', m.id_municipio) as string + ) as id_empenho_bd, + safe_cast(idempenho as string) as id_empenho, + safe_cast(nrempenho as string) as numero, + safe_cast(lower(dshistorico) as string) as descricao, + safe_cast(left(dstipoempenho, 1) as string) as modalidade, + safe_cast(safe_cast(cdfuncao as int64) as string) as funcao, + safe_cast(safe_cast(cdsubfuncao as int64) as string) as subfuncao, + safe_cast(safe_cast(cdprograma as int64) as string) as programa, + safe_cast(safe_cast(cdprojetoatividade as int64) as string) as acao, + safe_cast( + concat( + cdcategoriaeconomica, cdgruponatureza, cdmodalidade, cdelemento + ) as string + ) as elemento_despesa, + round(safe_cast(vlempenho as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_reforco, + round(safe_cast(vlestornoempenho as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round( + safe_cast(vlempenho as float64) + - ifnull(safe_cast(vlestornoempenho as float64), 0), + 2 + ) as valor_final + from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_pr") }} e + left join + basedosdados.br_bd_diretorios_brasil.municipio m + on e.cdibge = m.id_municipio_6 + + ) +select * +from empenho_pr diff --git a/models/world_wb_mides/states/world_wb_mides__empenho_rj.sql b/models/world_wb_mides/states/world_wb_mides__empenho_rj.sql new file mode 100644 index 0000000000..c02d531690 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__empenho_rj.sql @@ -0,0 +1,424 @@ +-- Rio de Janeiro (RJ) contribution to world_wb_mides.empenho. +-- Split out of the monolithic empenho model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + empenhado_municipio_rj_v1 as ( + select + (safe_cast(exercicio_empenho as int64)) as ano, + (safe_cast(extract(month from date(data_empenho)) as int64)) as mes, + safe_cast(data_empenho as date) as data, + 'RJ' as sigla_uf, + '3304557' as id_municipio, + safe_cast(orgao_programa_trabalho as string) as orgao, + safe_cast(unidade_programa_trabalho as string) as id_unidade_gestora, + safe_cast(null as string) as id_licitacao_bd, + safe_cast(n_mero_licita__o as string) as id_licitacao, + case + when modalidade_licitacao = 'Convite' + then '1' + when modalidade_licitacao = 'Tomada De Preços' + then '2' + when modalidade_licitacao = 'Tomada de Preços' + then '2' + when modalidade_licitacao = 'Concorrência' + then '3' + when modalidade_licitacao = 'Pregão' + then '4' + when modalidade_licitacao = 'Leilão' + then '7' + when modalidade_licitacao = 'Dispensa' + then '8' + when modalidade_licitacao = 'Inexigibilidade' + then '10' + when modalidade_licitacao = 'Concurso' + then '11' + when modalidade_licitacao = 'Seleção Pública' + then '31' + when modalidade_licitacao = 'Não Sujeito' + then '99' + end as modalidade_licitacao, + safe_cast( + concat( + nr_empenho, + ' ', + trim(orgao_programa_trabalho), + ' ', + trim(unidade_programa_trabalho), + ' ', + '3304557', + ' ', + (right(exercicio_empenho, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(nr_empenho as string) as numero, + safe_cast(null as string) as descricao, + safe_cast(left(especie, 1) as string) as modalidade, + safe_cast( + cast(substring(programa_trabalho, 7, 2) as int64) as string + ) as funcao, + safe_cast( + cast(substring(programa_trabalho, 10, 3) as int64) as string + ) as subfuncao, + safe_cast(substring(programa_trabalho, 14, 4) as string) as programa, + safe_cast(substring(programa_trabalho, 19, 4) as string) as acao, + safe_cast( + safe_cast(natureza_despesa as int64) as string + ) as modalidade_despesa, + round(safe_cast(valor_empenhado as float64), 2) as valor_final, + from + {{ + set_datalake_project( + "world_wb_mides_staging.raw_despesa_rj_municipio" + ) + }} + where (safe_cast(exercicio_empenho as int64)) < 2017 + + ), + frequencia_rj_v1 as ( + select id_empenho_bd, count(id_empenho_bd) as frequencia_id + from empenhado_municipio_rj_v1 + group by 1 + order by 2 desc + + ), + empenho_municipio_rj_v1 as ( + select + e.ano, + e.mes, + e.data, + e.sigla_uf, + e.id_municipio, + e.orgao, + e.id_unidade_gestora, + e.id_licitacao_bd, + e.id_licitacao, + e.modalidade_licitacao, + ( + case + when frequencia_id > 1 + then (safe_cast(null as string)) + else e.id_empenho_bd + end + ) as id_empenho_bd, + e.id_empenho, + e.numero, + e.descricao, + e.modalidade, + e.funcao, + e.subfuncao, + e.programa, + e.acao, + e.modalidade_despesa, + round(safe_cast(0 as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_reforco, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + e.valor_final as valor_final + from empenhado_municipio_rj_v1 e + left join frequencia_rj_v1 f on e.id_empenho_bd = f.id_empenho_bd + + ), + empenhado_municipio_rj_v2 as ( + select + (safe_cast(exercicio as int64)) as ano, + (safe_cast(extract(month from date(data)) as int64)) as mes, + safe_cast(data as date) as data, + 'RJ' as sigla_uf, + '3304557' as id_municipio, + safe_cast(ug as string) as orgao, + safe_cast(uo as string) as id_unidade_gestora, + safe_cast(null as string) as id_licitacao_bd, + safe_cast(null as string) as id_licitacao, + case + when licitacao = 'CONVITE' + then '1' + when licitacao = 'TOMADA DE PREÇOS' + then '2' + when licitacao = 'CONCORRÊNCIA' + then '3' + when licitacao = 'PREGÃO' + then '4' + when licitacao = 'PREÇO REGISTRADO/PREGÃO' + then '4' + when licitacao = 'REGISTRO DE PREÇOS EXTERNO/PREGÃO' + then '4' + when licitacao = 'DISPENSA' + then '8' + when licitacao = 'INEXIGIBILIDADE' + then '10' + when licitacao = 'CONCURSO' + then '11' + when licitacao = 'SELEÇÃO PÚBLICA' + then '31' + when licitacao = 'NÃO SUJEITO' + then '99' + end as modalidade_licitacao, + safe_cast( + concat( + left(empenhoexercicio, length(empenhoexercicio) - 5), + ' ', + trim(uo), + ' ', + trim(ug), + ' ', + '3304557', + ' ', + (right(exercicio, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(empenhoexercicio as string) as numero, + safe_cast(historico as string) as descricao, + safe_cast(null as string) as modalidade, + safe_cast(cast(funcao as int64) as string) as funcao, + safe_cast(subfuncao as string) as subfuncao, + safe_cast(programa as string) as programa, + safe_cast(acao as string) as acao, + safe_cast( + concat( + -- categoria econômica + case + when grupo = 'PESSOAL E ENCARGOS SOCIAIS' + then '3' + when grupo = 'JUROS E ENCARGOS DA DIVIDA' + then '3' + when grupo = 'OUTRAS DESPESAS CORRENTES' + then '3' + when grupo = 'INVESTIMENTOS' + then '4' + when grupo = 'INVERSOES FINANCEIRAS' + then '4' + when grupo = 'AMORTIZACAO DA DIVIDA' + then '4' + end, + -- natureza da despesa + case + when grupo = 'PESSOAL E ENCARGOS SOCIAIS' + then '1' + when grupo = 'JUROS E ENCARGOS DA DIVIDA' + then '2' + when grupo = 'OUTRAS DESPESAS CORRENTES' + then '3' + when grupo = 'INVESTIMENTOS' + then '4' + when grupo = 'INVERSOES FINANCEIRAS' + then '5' + when grupo = 'AMORTIZACAO DA DIVIDA' + then '6' + end, + -- modalidade de aplicação + case + when modalidade = 'TRANSFERENCIAS A UNIAO' + then '20' + when + modalidade + = 'TRANSFERENCIAS A ESTADOS E AO DISTRITO FEDERAL' + then '30' + when + modalidade + = 'TRANSFERENCIAS A INSTITUICOES PRIVADAS SEM FINS LUCRATIVOS' + then '50' + when + modalidade + = 'TRANSFERENCIAS A INSTITUICOES PRIVADAS COM FINS LUCRATIVOS' + then '60' + when + modalidade + = 'EXECUCAO DE CONTRATO DE PARCERIA PUBLICO-PRIVADA' + then '67' + when + modalidade + = 'EXECUCAO DE CONTRATO DE PARCERIA PUBLICO-PRIVADA - PPP' + then '67' + when + modalidade + = 'EXECUCAO DE CONTRATO DE PARCERIA PUBLICO PRIVADA - PPP' + then '67' + when + modalidade + = 'DESP. DECORRENTES DA PART. EM FUNDOS, ORGANISMOS OU ENTIDADES ASSEMELHADAS NAC. E INTERN.' + then '84' + when modalidade = 'APLICACOES DIRETAS' + then '90' + when + modalidade + = 'APLIC. DIRETA DECOR. DE OPER. ENTRE ORG., FUNDOS E ENTID. INTEG. DO ORC. FISC. E SEG. SOC.' + then '91' + when + modalidade + = 'APLIC DIRETAS DECOR DE OPER ENTRE ORG, FUNDOS E ENTID INTEGRANTES DOS ORC FISC E SEG SOC' + then '91' + else null + end, + -- elemento e item da despesa + elemento, + subelemento + ) as string + ) as elemento_despesa, + round(safe_cast(valor as float64), 2) as valor_inicial, + from + {{ + set_datalake_project( + "world_wb_mides_staging.raw_despesa_ato_rj_municipio" + ) + }} + where tipoato = 'EMPENHO' + + ), + anulacao_municipio_rj_v2 as ( + select + safe_cast( + concat( + left(empenhoexercicio, length(empenhoexercicio) - 5), + ' ', + trim(uo), + ' ', + trim(ug), + ' ', + '3304557', + ' ', + (right(exercicio, 2)) + ) as string + ) as id_empenho_bd, + round(sum(safe_cast(valor as float64)), 2) as valor_anulacao, + from + {{ + set_datalake_project( + "world_wb_mides_staging.raw_despesa_ato_rj_municipio" + ) + }} + where tipoato = 'CANCELAMENTO EMPENHO' + group by 1 + + ), + empenho_municipio_rj_v2 as ( + select + e.ano, + e.mes, + e.data, + e.sigla_uf, + e.id_municipio, + e.orgao, + e.id_unidade_gestora, + e.id_licitacao_bd, + e.id_licitacao, + e.modalidade_licitacao, + e.id_empenho_bd, + e.id_empenho, + e.numero, + e.descricao, + e.modalidade, + e.funcao, + e.subfuncao, + e.programa, + e.acao, + e.elemento_despesa, + e.valor_inicial as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_reforco, + safe_cast(ifnull(a.valor_anulacao, 0) as float64) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round( + safe_cast((e.valor_inicial - ifnull(a.valor_anulacao, 0)) as float64), 2 + ) as valor_final + from empenhado_municipio_rj_v2 e + left join anulacao_municipio_rj_v2 a on e.id_empenho_bd = a.id_empenho_bd + + ), + empenhado_rj as ( + select + (safe_cast(ano as int64)) as ano, + (safe_cast(extract(month from date(data)) as int64)) as mes, + safe_cast(data as date) as data, + 'RJ' as sigla_uf, + safe_cast(id_municipio as string) as id_municipio, + safe_cast(id_orgao as string) as orgao, + safe_cast(unidade_administrativa as string) as id_unidade_gestora, + safe_cast(null as string) as id_licitacao_bd, + safe_cast(null as string) as id_licitacao, + safe_cast(null as string) as modalidade_licitacao, + safe_cast( + concat( + numero_empenho, + ' ', + id_orgao, + ' ', + id_municipio, + ' ', + (right(ano, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(numero_empenho as string) as numero, + safe_cast(descricao as string) as descricao, + safe_cast(modalidade as string) as modalidade, + safe_cast(cast(funcao as int64) as string) as funcao, + safe_cast(subfuncao as string) as subfuncao, + safe_cast(programa as string) as programa, + safe_cast(atividade as string) as acao, + safe_cast(elemento_despesa as string) as elemento_despesa, + round(safe_cast(valor as float64), 2) as valor_inicial, + from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_rj") }} + where numero_empenho is not null + + ), + anulacao_rj as ( + select + safe_cast( + concat( + numero_empenho, + ' ', + id_orgao, + ' ', + id_municipio, + ' ', + (right(ano, 2)) + ) as string + ) as id_empenho_bd, + round(safe_cast(valor as float64), 2) as valor_anulacao, + from {{ set_datalake_project("world_wb_mides_staging.raw_anulacao_rj") }} + where despesa_liquidada = 'NÃO' and numero_empenho is not null + + ), + empenho_rj as ( + select + e.ano, + e.mes, + e.data, + e.sigla_uf, + e.id_municipio, + e.orgao, + e.id_unidade_gestora, + e.id_licitacao_bd, + e.id_licitacao, + e.modalidade_licitacao, + e.id_empenho_bd, + e.id_empenho, + e.numero, + e.descricao, + e.modalidade, + e.funcao, + e.subfuncao, + e.programa, + e.acao, + e.elemento_despesa, + e.valor_inicial as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_reforco, + safe_cast(ifnull(a.valor_anulacao, 0) as float64) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round( + safe_cast((e.valor_inicial - ifnull(a.valor_anulacao, 0)) as float64), 2 + ) as valor_final + from empenhado_rj e + left join anulacao_rj a on e.id_empenho_bd = a.id_empenho_bd + + ) +select * +from empenho_rj +union all +(select * from empenho_municipio_rj_v1) +union all +(select * from empenho_municipio_rj_v2) diff --git a/models/world_wb_mides/states/world_wb_mides__empenho_rs.sql b/models/world_wb_mides/states/world_wb_mides__empenho_rs.sql new file mode 100644 index 0000000000..773fc5288d --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__empenho_rs.sql @@ -0,0 +1,261 @@ +-- Rio Grande do Sul (RS) contribution to world_wb_mides.empenho. +-- Split out of the monolithic empenho model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + empenhado_rs as ( + select + min(ano_recebimento) as ano_recebimento, + safe_cast(ano_empenho as int64) as ano, + safe_cast(extract(month from date(dt_operacao)) as int64) as mes, + safe_cast( + concat( + substring(dt_operacao, 1, 4), + '-', + substring(dt_operacao, 6, 2), + '-', + substring(dt_operacao, 9, 2) + ) as date + ) as data, + 'RS' as sigla_uf, + safe_cast(a.id_municipio as string) as id_municipio, + safe_cast(c.cd_orgao as string) as orgao, + safe_cast(cd_orgao_orcamentario as string) as id_unidade_gestora, + safe_cast(null as string) as id_licitacao_bd, + safe_cast(null as string) as id_licitacao, + safe_cast(null as string) as modalidade_licitacao, + safe_cast( + concat( + nr_empenho, + ' ', + c.cd_orgao, + ' ', + m.id_municipio, + ' ', + (right(ano_empenho, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(nr_empenho as string) as numero, + safe_cast(lower(historico) as string) as descricao, + safe_cast(null as string) as modalidade, + safe_cast(safe_cast(cd_funcao as int64) as string) as funcao, + safe_cast(safe_cast(cd_subfuncao as int64) as string) as subfuncao, + safe_cast(safe_cast(cd_programa as int64) as string) as programa, + safe_cast(safe_cast(cd_projeto as int64) as string) as acao, + safe_cast(replace(cd_elemento, '.', '') as string) as elemento_despesa, + safe_cast(vl_empenho as float64) as valor_inicial + from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} as c + left join + {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} as a + on c.cd_orgao = a.cd_orgao + left join + `basedosdados.br_bd_diretorios_brasil.municipio` m + on m.id_municipio = a.id_municipio + where tipo_operacao = 'E' and (safe_cast(vl_empenho as float64) >= 0) + group by + 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22 + + ), + frequencia_rs as ( + select id_empenho_bd, count(id_empenho_bd) as frequencia_id + from empenhado_rs + group by 1 + + ), + anulacao_rs as ( + select + safe_cast( + concat( + nr_empenho, + ' ', + c.cd_orgao, + ' ', + m.id_municipio, + ' ', + (right(ano_empenho, 2)) + ) as string + ) as id_empenho_bd, + -1 * sum(safe_cast(vl_empenho as float64)) as valor_anulacao + from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} as c + left join + {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} as a + on c.cd_orgao = a.cd_orgao + left join + `basedosdados.br_bd_diretorios_brasil.municipio` m + on m.id_municipio = a.id_municipio + where tipo_operacao = 'E' and (safe_cast(vl_empenho as float64) < 0) + group by 1 + + ), + empenho_anulacao as ( + select + e.*, f.frequencia_id, a.valor_anulacao / f.frequencia_id as valor_anulacao + from empenhado_rs e + left join anulacao_rs a on e.id_empenho_bd = a.id_empenho_bd + left join frequencia_rs f on e.id_empenho_bd = f.id_empenho_bd + + ), + dorgao as ( + select + id_empenho_bd, + case when (count(distinct orgao)) > 1 then 1 else 0 end as dorgao + from empenho_anulacao + group by 1 + + ), + dugest as ( + select + id_empenho_bd, + case + when (count(distinct id_unidade_gestora)) > 1 then 1 else 0 + end as dugest + from empenho_anulacao + group by 1 + + ), + ddesc as ( + select + id_empenho_bd, + case when (count(distinct descricao)) > 1 then 1 else 0 end as ddesc + from empenho_anulacao + group by 1 + + ), + dfun as ( + select + id_empenho_bd, + case when (count(distinct funcao)) > 1 then 1 else 0 end as dfun + from empenho_anulacao + group by 1 + + ), + dsubf as ( + select + id_empenho_bd, + case when (count(distinct subfuncao)) > 1 then 1 else 0 end as dsubf + from empenho_anulacao + group by 1 + + ), + dprog as ( + select + id_empenho_bd, + case when (count(distinct programa)) > 1 then 1 else 0 end as dprog + from empenho_anulacao + group by 1 + + ), + dacao as ( + select + id_empenho_bd, + case when (count(distinct acao)) > 1 then 1 else 0 end as dacao + from empenho_anulacao + group by 1 + + ), + delem as ( + select + id_empenho_bd, + case when (count(distinct elemento_despesa)) > 1 then 1 else 0 end as delem + from empenho_anulacao + group by 1 + + ), + dummies as ( + select o.id_empenho_bd, dorgao, dugest, ddesc, dfun, dsubf, dprog, dacao, delem + from dorgao o + left join dugest g on o.id_empenho_bd = g.id_empenho_bd + left join ddesc d on o.id_empenho_bd = d.id_empenho_bd + left join dfun f on o.id_empenho_bd = f.id_empenho_bd + left join dsubf s on o.id_empenho_bd = s.id_empenho_bd + left join dprog p on o.id_empenho_bd = p.id_empenho_bd + left join dacao a on o.id_empenho_bd = a.id_empenho_bd + left join delem e on o.id_empenho_bd = e.id_empenho_bd + + ), + empenho_rs as ( + select + min(e.ano) as ano, + min(e.mes) as mes, + min(e.data) as data, + e.sigla_uf, + e.id_municipio, + e.orgao, + e.id_unidade_gestora, + e.id_licitacao_bd, + e.id_licitacao, + e.modalidade_licitacao, + ( + case + when + ( + dorgao = 1 + or dugest = 1 + or dfun = 1 + or dsubf = 1 + or dprog = 1 + or dacao = 1 + or delem = 1 + ) + then (safe_cast(null as string)) + else e.id_empenho_bd + end + ) as id_empenho_bd, + e.id_empenho, + e.numero, + ( + case + when + ( + ddesc = 1 + and ( + dorgao = 0 + or dugest = 0 + or dfun = 0 + or dsubf = 0 + or dprog = 0 + or dacao = 0 + or delem = 0 + ) + ) + then (safe_cast(null as string)) + when + ( + ddesc = 1 + and ( + dorgao = 1 + or dugest = 1 + or dfun = 1 + or dsubf = 1 + or dprog = 1 + or dacao = 1 + or delem = 1 + ) + ) + then (safe_cast(e.descricao as string)) + else e.descricao + end + ) as descricao, + e.modalidade, + e.funcao, + e.subfuncao, + e.programa, + e.acao, + e.elemento_despesa, + round(sum(e.valor_inicial), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_reforco, + round(sum(e.valor_anulacao), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round( + sum(e.valor_inicial) - ifnull(sum(e.valor_anulacao), 0), 2 + ) as valor_final + from empenho_anulacao e + left join dummies d on e.id_empenho_bd = d.id_empenho_bd + group by 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 + + ) +select * +from empenho_rs diff --git a/models/world_wb_mides/states/world_wb_mides__empenho_sc.sql b/models/world_wb_mides/states/world_wb_mides__empenho_sc.sql new file mode 100644 index 0000000000..c664de82c6 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__empenho_sc.sql @@ -0,0 +1,115 @@ +-- Santa Catarina (SC) contribution to world_wb_mides.empenho. +-- Split out of the monolithic empenho model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + empenhado_sc as ( + select + safe_cast(ano_emp as int64) as ano, + safe_cast(substring(trim(data_empenho), -7, 2) as int64) as mes, + safe_cast( + concat( + substring(trim(data_empenho), -4), + '-', + substring(trim(data_empenho), -7, 2), + '-', + substring(trim(data_empenho), 1, 2) + ) as date + ) as data, + 'SC' as sigla_uf, + safe_cast(id_municipio as string) as id_municipio, + safe_cast(codigo_orgao as string) as orgao, + safe_cast(null as string) as id_unidade_gestora, + safe_cast(null as string) as id_licitacao_bd, + safe_cast( + case + when + (split(nr_licitacao_contrato_convenio, ' / ')[offset(0)]) + != "Sem Licitação" + and (split(nr_licitacao_contrato_convenio, ' / ')[offset(0)]) + != "Sem licitação" + and (split(nr_licitacao_contrato_convenio, ' / ')[offset(0)]) + != "Sem Licitacao" + and (split(nr_licitacao_contrato_convenio, ' / ')[offset(0)]) + != "SEM LICITACAO" + then (split(nr_licitacao_contrato_convenio, ' / ')[offset(0)]) + else '' + end as string + ) as id_licitacao, + safe_cast(null as string) as modalidade_licitacao, + safe_cast( + concat( + num_empenho, + ' ', + codigo_orgao, + ' ', + id_municipio, + ' ', + (right(cast(ano_emp as string), 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(num_empenho as string) as numero, + safe_cast(lower(descricao_historico_empenho) as string) as descricao, + safe_cast(null as string) as modalidade, + safe_cast(cast(left(funcao, 2) as int64) as string) as funcao, + safe_cast(cast(left(subfuncao, 3) as int64) as string) as subfuncao, + safe_cast(null as string) as programa, + safe_cast(null as string) as acao, + safe_cast(elemento_despesa as string) as elemento_despesa, + round(safe_cast(valor_empenho as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_reforco, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(valor_empenho as float64), 2) as valor_final + from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_sc") }} + + ), + frequencia_sc as ( + select id_empenho_bd, count(id_empenho_bd) as frequencia_id + from empenhado_sc + group by 1 + order by 2 desc + + ), + empenho_sc as ( + select + e.ano, + e.mes, + e.data, + e.sigla_uf, + e.id_municipio, + e.orgao, + e.id_unidade_gestora, + e.id_licitacao_bd, + e.id_licitacao, + e.modalidade_licitacao, + ( + case + when frequencia_id > 1 + then (safe_cast(null as string)) + else e.id_empenho_bd + end + ) as id_empenho_bd, + e.id_empenho, + e.numero, + e.descricao, + e.modalidade, + e.funcao, + e.subfuncao, + e.programa, + e.acao, + e.elemento_despesa, + round(safe_cast(0 as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_reforco, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + e.valor_final as valor_final + from empenhado_sc e + left join frequencia_sc f on e.id_empenho_bd = f.id_empenho_bd + + ) +select * +from empenho_sc diff --git a/models/world_wb_mides/states/world_wb_mides__empenho_sp.sql b/models/world_wb_mides/states/world_wb_mides__empenho_sp.sql new file mode 100644 index 0000000000..ca97c928d6 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__empenho_sp.sql @@ -0,0 +1,363 @@ +-- São Paulo (SP) contribution to world_wb_mides.empenho. +-- Split out of the monolithic empenho model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + empenhado_sp as ( + select + safe_cast(ano_exercicio as int64) as ano, + safe_cast(mes_referencia as int64) as mes, + safe_cast( + concat( + substring(dt_emissao_despesa, -4), + '-', + substring(dt_emissao_despesa, -7, 2), + '-', + substring(dt_emissao_despesa, 1, 2) + ) as date + ) as data, + 'SP' as sigla_uf, + safe_cast(id_municipio as string) as id_municipio, + safe_cast(codigo_orgao as string) as orgao, + safe_cast(null as string) as id_unidade_gestora, + safe_cast(null as string) as id_licitacao_bd, + safe_cast(null as string) as id_licitacao, + case + when ds_modalidade_lic = 'CONVITE' + then '1' + when ds_modalidade_lic = 'TOMADA DE PREÇOS' + then '2' + when ds_modalidade_lic = 'CONCORRÊNCIA' + then '3' + when ds_modalidade_lic = 'PREGÃO' + then '4' + when ds_modalidade_lic = 'Leilão' + then '7' + when ds_modalidade_lic = 'DISPENSA DE LICITAÇÃO' + then '8' + when ds_modalidade_lic = 'BEC-BOLSA ELETRÔNICA DE COMPRAS' + then '9' + when ds_modalidade_lic = 'INEXIGÍVEL' + then '10' + when ds_modalidade_lic = 'CONCURSO' + then '11' + when ds_modalidade_lic = 'RDC' + then '12' + when ds_modalidade_lic = 'OUTROS/NÃO APLICÁVEL' + then '99' + end as modalidade_licitacao, + safe_cast( + concat( + left(nr_empenho, length(nr_empenho) - 5), + ' ', + codigo_orgao, + ' ', + id_municipio, + ' ', + (right(ano_exercicio, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(nr_empenho as string) as numero, + safe_cast(lower(historico_despesa) as string) as descricao, + safe_cast(null as string) as modalidade, + safe_cast(safe_cast(funcao as int64) as string) as funcao, + safe_cast(safe_cast(subfuncao as int64) as string) as subfuncao, + safe_cast(safe_cast(cd_programa as int64) as string) as programa, + safe_cast(safe_cast(cd_acao as int64) as string) as acao, + safe_cast((left(ds_elemento, 8)) as string) as elemento_despesa, + safe_cast(replace(vl_despesa, ',', '.') as float64) as valor_inicial + from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_sp") }} e + left join + {{ set_datalake_project("world_wb_mides_staging.aux_municipio_sp") }} m + on m.ds_orgao = e.ds_orgao + left join + {{ set_datalake_project("world_wb_mides_staging.aux_funcao") }} + on ds_funcao_governo = upper(nome_funcao) + left join + {{ set_datalake_project("world_wb_mides_staging.aux_subfuncao") }} + on ds_subfuncao_governo = upper(nome_subfuncao) + where tp_despesa = 'Empenhado' + + ), + frequencia_sp as ( + select id_empenho_bd, count(id_empenho_bd) as frequencia_id + from empenhado_sp + group by 1 + order by 2 desc + + ), + anulacao as ( + select + safe_cast( + concat( + left(nr_empenho, length(nr_empenho) - 5), + ' ', + codigo_orgao, + ' ', + id_municipio, + ' ', + (right(ano_exercicio, 2)) + ) as string + ) as id_empenho_bd, + sum(safe_cast(replace(vl_despesa, ',', '.') as float64)) as valor_anulacao + from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_sp") }} a + left join + {{ set_datalake_project("world_wb_mides_staging.aux_municipio_sp") }} m + on m.ds_orgao = a.ds_orgao + where tp_despesa = 'Anulação' + group by 1 + + ), + reforco as ( + select + safe_cast( + concat( + left(nr_empenho, length(nr_empenho) - 5), + ' ', + codigo_orgao, + ' ', + id_municipio, + ' ', + (right(ano_exercicio, 2)) + ) as string + ) as id_empenho_bd, + sum(safe_cast(replace(vl_despesa, ',', '.') as float64)) as valor_reforco + from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_sp") }} r + left join + {{ set_datalake_project("world_wb_mides_staging.aux_municipio_sp") }} m + on m.ds_orgao = r.ds_orgao + where tp_despesa = 'Reforço' + group by 1 + + ), + empenho_completo_sp as ( + select + e.*, + r.valor_reforco / frequencia_id as valor_reforco, + a.valor_anulacao / frequencia_id as valor_anulacao, + from empenhado_sp e + left join frequencia_sp f on e.id_empenho_bd = f.id_empenho_bd + left join anulacao a on e.id_empenho_bd = a.id_empenho_bd + left join reforco r on e.id_empenho_bd = r.id_empenho_bd + + ), + dorgao_sp as ( + select + id_empenho_bd, + case when (count(distinct orgao)) > 1 then 1 else 0 end as dorgao + from empenho_completo_sp + group by 1 + + ), + ddesc_sp as ( + select + id_empenho_bd, + case + when (count(distinct ifnull(descricao, ''))) > 1 then 1 else 0 + end as ddesc + from empenho_completo_sp + group by 1 + + ), + dmod_sp as ( + select + id_empenho_bd, + case + when (count(distinct modalidade_licitacao)) > 1 then 1 else 0 + end as dmod + from empenho_completo_sp + group by 1 + + ), + dfun_sp as ( + select + id_empenho_bd, + case when (count(distinct funcao)) > 1 then 1 else 0 end as dfun + from empenho_completo_sp + group by 1 + + ), + dsubf_sp as ( + select + id_empenho_bd, + case when (count(distinct subfuncao)) > 1 then 1 else 0 end as dsubf + from empenho_completo_sp + group by 1 + + ), + dprog_sp as ( + select + id_empenho_bd, + case when (count(distinct programa)) > 1 then 1 else 0 end as dprog + from empenho_completo_sp + group by 1 + + ), + dacao_sp as ( + select + id_empenho_bd, + case when (count(distinct acao)) > 1 then 1 else 0 end as dacao + from empenho_completo_sp + group by 1 + + ), + delem_sp as ( + select + id_empenho_bd, + case when (count(distinct elemento_despesa)) > 1 then 1 else 0 end as delem + from empenho_completo_sp + group by 1 + + ), + dummies_sp as ( + select o.id_empenho_bd, dorgao, dmod, ddesc, dfun, dsubf, dprog, dacao, delem + from dorgao_sp o + full outer join dmod_sp m on o.id_empenho_bd = m.id_empenho_bd + full outer join ddesc_sp d on o.id_empenho_bd = d.id_empenho_bd + full outer join dfun_sp f on o.id_empenho_bd = f.id_empenho_bd + full outer join dsubf_sp s on o.id_empenho_bd = s.id_empenho_bd + full outer join dprog_sp p on o.id_empenho_bd = p.id_empenho_bd + full outer join dacao_sp a on o.id_empenho_bd = a.id_empenho_bd + full outer join delem_sp e on o.id_empenho_bd = e.id_empenho_bd + + ), + empenho_sp as ( + select + min(ano) as ano, + min(mes) as mes, + min(data) as data, + sigla_uf, + id_municipio, + orgao, + id_unidade_gestora, + id_licitacao_bd, + id_licitacao, + modalidade_licitacao, + ( + case + when + ( + dorgao = 1 + or dmod = 1 + or dfun = 1 + or dsubf = 1 + or dprog = 1 + or dacao = 1 + or delem = 1 + ) + then (safe_cast(null as string)) + else e.id_empenho_bd + end + ) as id_empenho_bd, + id_empenho, + numero, + case + when + ( + ddesc = 1 + and ( + dorgao = 0 + or dmod = 0 + or dfun = 0 + or dsubf = 0 + or dprog = 0 + or dacao = 0 + or delem = 0 + ) + ) + then (safe_cast(null as string)) + when + ( + ddesc = 1 + and ( + dorgao = 1 + or dmod = 1 + or dfun = 1 + or dsubf = 1 + or dprog = 1 + or dacao = 1 + or delem = 1 + ) + ) + then (safe_cast(e.descricao as string)) + else e.descricao + end as descricao, + modalidade, + funcao, + subfuncao, + programa, + acao, + elemento_despesa, + round(sum(valor_inicial), 2) as valor_inicial, + round(sum(valor_reforco), 2) as valor_reforco, + round(sum(valor_anulacao), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round( + ifnull(sum(valor_inicial), 0) + + ifnull(sum(valor_reforco), 0) + - ifnull(sum(valor_anulacao), 0), + 2 + ) as valor_final + from empenho_completo_sp e + left join dummies_sp d on d.id_empenho_bd = e.id_empenho_bd + group by 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 + + ), + empenho_municipio_sp as ( + select + (safe_cast(exercicio as int64)) as ano, + (safe_cast(extract(month from date(data_empenho)) as int64)) as mes, + safe_cast(data_empenho as date) as data, + 'SP' as sigla_uf, + '3550308' as id_municipio, + safe_cast(codigo_orgao as string) as orgao, + safe_cast(codigo_unidade as string) as id_unidade_gestora, + safe_cast(null as string) as id_licitacao_bd, + safe_cast(null as string) as id_licitacao, + safe_cast(null as string) as modalidade_licitacao, + safe_cast( + concat( + nr_empenho, + ' ', + trim(codigo_orgao), + ' ', + trim(codigo_unidade), + ' ', + '3550308', + ' ', + (right(exercicio, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(id_empenho as string) as id_empenho, + safe_cast(nr_empenho as string) as numero, + safe_cast(observacoes as string) as descricao, + safe_cast( + left(replace(tipo_empenho, 'Por Estimativa', 'Estimativo'), 1) as string + ) as modalidade, + safe_cast(codigo_funcao as string) as funcao, + safe_cast(codigo_subfuncao as string) as subfuncao, + safe_cast(codigo_programa_governo as string) as programa, + safe_cast(codigo_projeto_atividade as string) as acao, + safe_cast(codigo_conta_despesa as string) as modalidade_despesa, + round(safe_cast(valor_empenho as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_reforco, + round(safe_cast(cancelado as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round( + safe_cast(valor_empenho as float64) - safe_cast(cancelado as float64), 2 + ) as valor_final, + from + {{ + set_datalake_project( + "world_wb_mides_staging.raw_despesa_sp_municipio" + ) + }} + + ) +select * +from empenho_sp +union all +(select * from empenho_municipio_sp) diff --git a/models/world_wb_mides/states/world_wb_mides__liquidacao_ce.sql b/models/world_wb_mides/states/world_wb_mides__liquidacao_ce.sql new file mode 100644 index 0000000000..4dc0172766 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__liquidacao_ce.sql @@ -0,0 +1,51 @@ +-- Ceará (CE) contribution to world_wb_mides.liquidacao. +-- Split out of the monolithic liquidacao model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + liquidacao_ce as ( + select + (safe_cast(extract(year from date(data_liquidacao)) as int64)) as ano, + (safe_cast(extract(month from date(data_liquidacao)) as int64)) as mes, + safe_cast(extract(date from timestamp(data_liquidacao)) as date) as data, + 'CE' as sigla_uf, + safe_cast(geoibgeid as string) as id_municipio, + safe_cast(codigo_orgao as string) as orgao, + safe_cast(codigo_unidade as string) as id_unidade_gestora, + safe_cast( + concat( + numero_empenho, + ' ', + trim(codigo_orgao), + ' ', + trim(codigo_unidade), + ' ', + geoibgeid, + ' ', + (substring(data_emissao_empenho, 6, 2)), + ' ', + (substring(data_emissao_empenho, 3, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(numero_empenho as string) as numero_empenho, + safe_cast(null as string) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(null as string) as numero, + safe_cast(nome_responsavel_liquidacao as string) as nome_responsavel, + safe_cast(cpf_responsavel_liquidacao_ as string) as documento_responsavel, + safe_cast(null as bool) as indicador_restos_pagar, + round(safe_cast(valor_liquidado as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(valor_liquidado as float64), 2) as valor_final, + from {{ set_datalake_project("world_wb_mides_staging.raw_liquidacao_ce") }} l + left join + {{ set_datalake_project("world_wb_mides_staging.aux_municipio_ce") }} m + on l.codigo_municipio = m.codigo_municipio + + ) +select * +from liquidacao_ce diff --git a/models/world_wb_mides/states/world_wb_mides__liquidacao_df.sql b/models/world_wb_mides/states/world_wb_mides__liquidacao_df.sql new file mode 100644 index 0000000000..1fcbe7e2ac --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__liquidacao_df.sql @@ -0,0 +1,62 @@ +-- Distrito Federal (DF) contribution to world_wb_mides.liquidacao. +-- Split out of the monolithic liquidacao model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + liquidacao_df as ( + select + (safe_cast(exercicio as int64)) as ano, + (safe_cast(extract(month from date(emissao)) as int64)) as mes, + safe_cast(emissao as date) as data, + 'DF' as sigla_uf, + '5300108' as id_municipio, + safe_cast(codigo_ug as string) as orgao, + safe_cast(codigo_gestao as string) as id_unidade_gestora, + safe_cast( + concat( + right(nota_empenho, length(nota_empenho) - 6), + ' ', + codigo_ug, + ' ', + codigo_gestao, + ' ', + '5300108', + ' ', + (right(exercicio, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(nota_empenho as string) as numero_empenho, + case + when length(nota_lancamento) = 11 + then + safe_cast( + concat( + right(nota_lancamento, length(nota_lancamento) - 6), + ' ', + codigo_ug, + ' ', + codigo_gestao, + ' ', + '5300108', + ' ', + (right(exercicio, 2)) + ) as string + ) + end as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(nota_lancamento as string) as numero, + safe_cast(credor as string) as nome_responsavel, + safe_cast(cnpj_cpf_credor as string) as documento_responsavel, + safe_cast(null as bool) as indicador_restos_pagar, + round(safe_cast(0 as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(replace(valor, ',', '.') as float64), 2) as valor_inicial, + from {{ set_datalake_project("world_wb_mides_staging.raw_liquidacao_df") }} + + ) +select * +from liquidacao_df diff --git a/models/world_wb_mides/states/world_wb_mides__liquidacao_mg.sql b/models/world_wb_mides/states/world_wb_mides__liquidacao_mg.sql new file mode 100644 index 0000000000..2afaae6eda --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__liquidacao_mg.sql @@ -0,0 +1,87 @@ +-- Minas Gerais (MG) contribution to world_wb_mides.liquidacao. +-- Split out of the monolithic liquidacao model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + liquidacao_mg as ( + select + safe_cast(ano as int64) as ano, + safe_cast(mes as int64) as mes, + safe_cast(data as date) as data, + 'MG' as sigla_uf, + safe_cast(l.id_municipio as string) as id_municipio, + safe_cast(l.orgao as string) as orgao, + safe_cast(l.id_unidade_gestora as string) as id_unidade_gestora, + safe_cast( + ( + case + when id_empenho != '-1' + then + concat( + id_empenho, + ' ', + l.orgao, + ' ', + l.id_municipio, + ' ', + (right(ano, 2)) + ) + when id_empenho = '-1' + then + concat( + id_empenho_origem, + ' ', + r.orgao, + ' ', + r.id_municipio, + ' ', + (right(num_ano_emp_origem, 2)) + ) + end + ) as string + ) as id_empenho_bd, + safe_cast( + ( + case + when id_empenho = '-1' + then replace (id_empenho, '-1', id_empenho_origem) + end + ) as string + ) as id_empenho, + safe_cast(numero_empenho as string) as numero_empenho, + safe_cast( + concat( + id_liquidacao, + ' ', + l.orgao, + ' ', + l.id_municipio, + ' ', + (right(ano, 2)) + ) as string + ) as id_liquidacao_bd, + safe_cast(id_liquidacao as string) as id_liquidacao, + safe_cast(numero_liquidacao as string) as numero, + safe_cast(nome_responsavel as string) as nome_responsavel, + safe_cast(documento_responsavel as string) as documento_responsavel, + safe_cast( + (case when l.id_rsp != '-1' then 1 else 0 end) as bool + ) as indicador_restos_pagar, + round(safe_cast(valor_liquidacao_original as float64), 2) as valor_inicial, + round(safe_cast(valor_anulado as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round( + safe_cast(valor_liquidacao_original as float64) + - ifnull(safe_cast(valor_anulado as float64), 0), + 2 + ) as valor_final + from {{ set_datalake_project("world_wb_mides_staging.raw_liquidacao_mg") }} as l + left join + {{ set_datalake_project("world_wb_mides_staging.raw_rsp_mg") }} as r + on l.id_rsp = r.id_rsp + + ) +select * +from liquidacao_mg diff --git a/models/world_wb_mides/states/world_wb_mides__liquidacao_pb.sql b/models/world_wb_mides/states/world_wb_mides__liquidacao_pb.sql new file mode 100644 index 0000000000..37b0c8b5f2 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__liquidacao_pb.sql @@ -0,0 +1,55 @@ +-- Paraíba (PB) contribution to world_wb_mides.liquidacao. +-- Split out of the monolithic liquidacao model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + liquidacao_pb as ( + select + safe_cast(dt_ano as int64) as ano, + (safe_cast(substring(dt_liquidacao, -7, 2) as int64)) as mes, + safe_cast( + concat( + substring(dt_liquidacao, -4), + '-', + substring(dt_liquidacao, -7, 2), + '-', + substring(dt_liquidacao, 1, 2) + ) as date + ) as data, + 'PB' as sigla_uf, + safe_cast(id_municipio as string) as id_municipio, + safe_cast(null as string) as orgao, + safe_cast(l.cd_ugestora as string) as id_unidade_gestora, + safe_cast( + concat( + nu_empenho, + ' ', + l.cd_ugestora, + ' ', + m.id_municipio, + ' ', + (right(dt_ano, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(nu_empenho as string) as numero_empenho, + safe_cast(null as string) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(nu_liquidacao as string) as numero, + safe_cast(null as string) as nome_responsavel, + safe_cast(null as string) as documento_responsavel, + safe_cast(null as bool) as indicador_restos_pagar, + round(safe_cast(vl_liquidacao as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(vl_liquidacao as float64), 2) as valor_final, + from {{ set_datalake_project("world_wb_mides_staging.raw_liquidacao_pb") }} l + left join + {{ set_datalake_project("world_wb_mides_staging.aux_municipio_pb") }} m + on l.cd_ugestora = safe_cast(m.id_unidade_gestora as string) + + ) +select * +from liquidacao_pb diff --git a/models/world_wb_mides/states/world_wb_mides__liquidacao_pe.sql b/models/world_wb_mides/states/world_wb_mides__liquidacao_pe.sql new file mode 100644 index 0000000000..743804c642 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__liquidacao_pe.sql @@ -0,0 +1,37 @@ +-- Pernambuco (PE) contribution to world_wb_mides.liquidacao. +-- Split out of the monolithic liquidacao model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + liquidacao_pe as ( + select + safe_cast(l.anoreferencia as int64) as ano, + (safe_cast(extract(month from date(data)) as int64)) as mes, + safe_cast(extract(date from timestamp(data)) as date) as data, + 'PE' as sigla_uf, + safe_cast(codigoibge as string) as id_municipio, + safe_cast(null as string) orgao, + safe_cast(id_unidadegestora as string) as id_unidade_gestora, + safe_cast(null as string) as id_empenho_bd, + safe_cast(trim(idempenho) as string) as id_empenho, + safe_cast(l.numeroempenho as string) as numero_empenho, + safe_cast(null as string) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(null as string) as numero, + safe_cast(null as string) as nome_responsavel, + safe_cast(null as string) as documento_responsavel, + safe_cast(null as bool) as indicador_restos_pagar, + round(safe_cast(valor as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(valor as float64), 2) as valor_final, + from {{ set_datalake_project("world_wb_mides_staging.raw_liquidacao_pe") }} l + left join + {{ set_datalake_project("world_wb_mides_staging.aux_municipio_pe") }} m + on l.id_unidade_gestora = safe_cast(m.id_unidadegestora as string) + + ) +select * +from liquidacao_pe diff --git a/models/world_wb_mides/states/world_wb_mides__liquidacao_pr.sql b/models/world_wb_mides/states/world_wb_mides__liquidacao_pr.sql new file mode 100644 index 0000000000..f390f6f2da --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__liquidacao_pr.sql @@ -0,0 +1,43 @@ +-- Paraná (PR) contribution to world_wb_mides.liquidacao. +-- Split out of the monolithic liquidacao model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + liquidacao_pr as ( + select + safe_cast(nranoliquidacao as int64) as ano, + (safe_cast(extract(month from date(dtliquidacao)) as int64)) as mes, + safe_cast(extract(date from timestamp(dtliquidacao)) as date) as data, + 'PR' as sigla_uf, + safe_cast(id_municipio as string) as id_municipio, + safe_cast(cdorgao as string) as orgao, + safe_cast(cdunidade as string) as id_unidade_gestora, + safe_cast( + concat(l.idempenho, ' ', m.id_municipio) as string + ) as id_empenho_bd, + safe_cast(l.idempenho as string) as id_empenho, + safe_cast(nrempenho as string) as numero_empenho, + safe_cast( + concat(l.idliquidacao, ' ', m.id_municipio) as string + ) as id_liquidacao_bd, + safe_cast(idliquidacao as string) as id_liquidacao, + safe_cast(nrliquidacao as string) as numero, + safe_cast(nmliquidante as string) as nome_responsavel, + safe_cast(nrdocliquidante as string) as documento_responsavel, + safe_cast(null as bool) as indicador_restos_pagar, + round(safe_cast(vlliquidacaobruto as float64), 2) as valor_inicial, + round(safe_cast(vlliquidacaoestornado as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(vlliquidacaoliquido as float64), 2) as valor_final, + from {{ set_datalake_project("world_wb_mides_staging.raw_liquidacao_pr") }} l + left join + basedosdados.br_bd_diretorios_brasil.municipio m on cdibge = id_municipio_6 + left join + {{ set_datalake_project("world_wb_mides_staging.raw_empenho_pr") }} e + on l.idempenho = e.idempenho + + ) +select * +from liquidacao_pr diff --git a/models/world_wb_mides/states/world_wb_mides__liquidacao_rj.sql b/models/world_wb_mides/states/world_wb_mides__liquidacao_rj.sql new file mode 100644 index 0000000000..02c93e1039 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__liquidacao_rj.sql @@ -0,0 +1,256 @@ +-- Rio de Janeiro (RJ) contribution to world_wb_mides.liquidacao. +-- Split out of the monolithic liquidacao model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + liquidado_municipio_rj_v1 as ( + select + safe_cast(exercicio_empenho as int64) as ano, + safe_cast(null as int64) as mes, + safe_cast(null as date) as data, + 'RJ' as sigla_uf, + '3304557' as id_municipio, + safe_cast(orgao_programa_trabalho as string) as orgao, + safe_cast(unidade_programa_trabalho as string) as id_unidade_gestora, + safe_cast( + concat( + nr_empenho, + ' ', + trim(orgao_programa_trabalho), + ' ', + trim(unidade_programa_trabalho), + ' ', + '3304557', + ' ', + (right(exercicio_empenho, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(nr_empenho as string) as numero_empenho, + safe_cast(null as string) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(null as string) as numero, + safe_cast(null as string) as nome_responsavel, + safe_cast(null as string) as documento_responsavel, + safe_cast(null as bool) as indicador_restos_pagar, + round(safe_cast(0 as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(valor_liquidado as float64), 2) as valor_final + from + {{ + set_datalake_project( + "world_wb_mides_staging.raw_despesa_rj_municipio" + ) + }} + where (safe_cast(exercicio_empenho as int64)) < 2017 + + ), + frequencia_rj_v1 as ( + select id_empenho_bd, count(id_empenho_bd) as frequencia_id + from liquidado_municipio_rj_v1 + group by 1 + order by 2 desc + + ), + liquidacao_municipio_rj_v1 as ( + select + l.ano, + l.mes, + l.data, + l.sigla_uf, + l.id_municipio, + l.orgao, + l.id_unidade_gestora, + ( + case + when frequencia_id > 1 + then (safe_cast(null as string)) + else l.id_empenho_bd + end + ) as id_empenho_bd, + l.id_empenho, + l.numero_empenho, + l.id_liquidacao_bd, + l.id_liquidacao, + l.numero, + l.nome_responsavel, + l.documento_responsavel, + l.indicador_restos_pagar, + l.valor_inicial, + l.valor_anulacao, + l.valor_ajuste, + l.valor_final + from liquidado_municipio_rj_v1 l + left join frequencia_rj_v1 f on l.id_empenho_bd = f.id_empenho_bd + + ), + liquidado_municipio_rj_v2 as ( + select + (safe_cast(exercicio as int64)) as ano, + (safe_cast(extract(month from date(data)) as int64)) as mes, + safe_cast(data as date) as data, + 'RJ' as sigla_uf, + '3304557' as id_municipio, + safe_cast(ug as string) as orgao, + safe_cast(uo as string) as id_unidade_gestora, + safe_cast( + concat( + left(empenhoexercicio, length(empenhoexercicio) - 5), + ' ', + trim(uo), + ' ', + trim(ug), + ' ', + '3304557', + ' ', + (right(exercicio, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(empenhoexercicio as string) as numero_empenho, + safe_cast( + concat( + liquidacao, + ' ', + left(empenhoexercicio, length(empenhoexercicio) - 5), + ' ', + trim(uo), + ' ', + trim(ug), + ' ', + '3304557', + ' ', + (right(exercicio, 2)) + ) as string + ) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(liquidacao as string) as numero, + safe_cast(null as string) as nome_responsavel, + safe_cast(null as string) as documento_responsavel, + safe_cast(null as bool) as indicador_restos_pagar, + round(safe_cast(valor as float64), 2) as valor_inicial + from + {{ + set_datalake_project( + "world_wb_mides_staging.raw_despesa_ato_rj_municipio" + ) + }} + where tipoato = 'LIQUIDACAO' + + ), + anulacao_municipio_rj_v2 as ( + select + safe_cast(tipoato as string) as tipoato, + safe_cast( + concat( + left(empenhoexercicio, length(empenhoexercicio) - 5), + ' ', + trim(uo), + ' ', + trim(ug), + ' ', + '3304557', + ' ', + (right(exercicio, 2)) + ) as string + ) as id_empenho_bd, + sum(safe_cast(valor as float64)) as valor_anulacao, + from + {{ + set_datalake_project( + "world_wb_mides_staging.raw_despesa_ato_rj_municipio" + ) + }} + where + tipoato in ( + 'CANCELAMENTO LIQUIDACAO', + 'Cancelamento de liquidação de RPN', + 'CANCELAMENTO DE RPN' + ) + group by 1, 2 + + ), + frequencia_rj_v2 as ( + select id_empenho_bd, count(1) as frequencia + from anulacao_municipio_rj_v2 + group by 1 + + ), + liquidacao_municipio_rj_v2 as ( + select + l.ano, + l.mes, + l.data, + l.sigla_uf, + l.id_municipio, + l.orgao, + l.id_unidade_gestora, + l.id_empenho_bd, + l.id_empenho, + l.numero_empenho, + l.id_liquidacao_bd, + l.id_liquidacao, + l.numero, + l.nome_responsavel, + l.documento_responsavel, + case + when tipoato = 'Cancelamento de liquidação de RPN' + then true + when tipoato = 'CANCELAMENTO DE RPN' + then true + else false + end as indicador_restos_pagar, + round(safe_cast(l.valor_inicial as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(l.valor_inicial as float64), 2) as valor_final + from liquidado_municipio_rj_v2 l + left join anulacao_municipio_rj_v2 a on l.id_empenho_bd = a.id_empenho_bd + left join frequencia_rj_v2 f on l.id_empenho_bd = f.id_empenho_bd + + ), + liquidacao_rj as ( + select + (safe_cast(ano as int64)) as ano, + (safe_cast(extract(month from date(data)) as int64)) as mes, + safe_cast(data as date) as data, + 'RJ' as sigla_uf, + safe_cast(id_municipio as string) as id_municipio, + safe_cast(id_orgao as string) as orgao, + safe_cast(unidade_administrativa as string) as id_unidade_gestora, + safe_cast( + concat( + numero_empenho, + ' ', + id_orgao, + ' ', + id_municipio, + ' ', + (right(ano, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(numero_empenho as string) as numero_empenho, + safe_cast(null as string) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(null as string) as numero, + safe_cast(null as string) as nome_responsavel, + safe_cast(null as string) as documento_responsavel, + safe_cast(null as bool) as indicador_restos_pagar, + round(safe_cast(valor as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(valor as float64), 2) as valor_final + from {{ set_datalake_project("world_wb_mides_staging.raw_liquidacao_rj") }} + where numero_empenho is not null + + ) +select * +from liquidacao_municipio_rj_v1 +union all +(select * from liquidacao_municipio_rj_v2) +union all +(select * from liquidacao_rj) diff --git a/models/world_wb_mides/states/world_wb_mides__liquidacao_rs.sql b/models/world_wb_mides/states/world_wb_mides__liquidacao_rs.sql new file mode 100644 index 0000000000..cbdea28a57 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__liquidacao_rs.sql @@ -0,0 +1,165 @@ +-- Rio Grande do Sul (RS) contribution to world_wb_mides.liquidacao. +-- Split out of the monolithic liquidacao model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + liquidado_rs as ( + select + min(ano_recebimento) as ano_recebimento, + safe_cast(ano_operacao as int64) as ano, + safe_cast(extract(month from date(dt_operacao)) as int64) as mes, + safe_cast( + concat( + substring(dt_operacao, 1, 4), + '-', + substring(dt_operacao, 6, 2), + '-', + substring(dt_operacao, 9, 2) + ) as date + ) as data, + 'RS' as sigla_uf, + safe_cast(a.id_municipio as string) as id_municipio, + safe_cast(c.cd_orgao as string) as orgao, + safe_cast(cd_orgao_orcamentario as string) as id_unidade_gestora, + safe_cast( + concat( + nr_empenho, + ' ', + c.cd_orgao, + ' ', + m.id_municipio, + ' ', + (right(ano_empenho, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(nr_empenho as string) as numero_empenho, + safe_cast( + concat( + nr_liquidacao, + ' ', + c.cd_orgao, + ' ', + m.id_municipio, + ' ', + (right(ano_empenho, 2)) + ) as string + ) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(nr_liquidacao as string) as numero, + safe_cast(null as string) as nome_responsavel, + safe_cast(null as string) as documento_responsavel, + safe_cast(null as bool) as indicador_restos_pagar, + safe_cast(vl_liquidacao as float64) as valor_inicial + from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} as c + left join + {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} as a + on c.cd_orgao = a.cd_orgao + left join + `basedosdados.br_bd_diretorios_brasil.municipio` m + on m.id_municipio = a.id_municipio + where tipo_operacao = 'L' and (safe_cast(vl_liquidacao as float64) >= 0) + group by 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18 + + ), + estorno_rs as ( + select + safe_cast( + concat( + nr_empenho, + ' ', + c.cd_orgao, + ' ', + m.id_municipio, + ' ', + (right(ano_empenho, 2)) + ) as string + ) as id_empenho_bd, + -1 * sum(safe_cast(vl_liquidacao as float64)) as valor_anulacao + from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} as c + left join + {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} as a + on c.cd_orgao = a.cd_orgao + left join + `basedosdados.br_bd_diretorios_brasil.municipio` m + on m.id_municipio = a.id_municipio + where tipo_operacao = 'L' and (safe_cast(vl_liquidacao as float64) < 0) + group by 1 + + ), + frequencia_rs as ( + select id_empenho_bd, count(id_empenho_bd) as frequencia_id + from liquidado_rs + group by 1 + + ), + liquidacao1_rs as ( + select + ano, + mes, + data, + sigla_uf, + id_municipio, + orgao, + id_unidade_gestora, + l.id_empenho_bd, + id_empenho, + numero_empenho, + id_liquidacao_bd, + id_liquidacao, + numero, + nome_responsavel, + documento_responsavel, + indicador_restos_pagar, + sum(valor_inicial) as valor_inicial, + sum(valor_anulacao / frequencia_id) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + sum( + valor_inicial - ifnull((valor_anulacao / frequencia_id), 0) + ) as valor_final + from liquidado_rs l + left join estorno_rs e on l.id_empenho_bd = e.id_empenho_bd + left join frequencia_rs f on l.id_empenho_bd = f.id_empenho_bd + group by 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16 + + ), + data_rs as ( + select + id_liquidacao_bd, + case when (count(distinct data)) > 1 then 1 else 0 end as ddata + from liquidacao1_rs + group by 1 + + ), + liquidacao_rs as ( + select + ano, + mes, + data, + sigla_uf, + id_municipio, + orgao, + id_unidade_gestora, + id_empenho_bd, + id_empenho, + numero_empenho, + case + when ddata = 1 then (safe_cast(null as string)) else l.id_liquidacao_bd + end as id_liquidacao_bd, + id_liquidacao, + numero, + nome_responsavel, + documento_responsavel, + indicador_restos_pagar, + round(valor_inicial, 2), + round(ifnull(valor_anulacao, 0), 2), + valor_ajuste, + round(valor_final, 2) + from liquidacao1_rs l + left join data_rs d on l.id_liquidacao_bd = d.id_liquidacao_bd + + ) +select * +from liquidacao_rs diff --git a/models/world_wb_mides/states/world_wb_mides__liquidacao_sc.sql b/models/world_wb_mides/states/world_wb_mides__liquidacao_sc.sql new file mode 100644 index 0000000000..1961fe7c5a --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__liquidacao_sc.sql @@ -0,0 +1,84 @@ +-- Santa Catarina (SC) contribution to world_wb_mides.liquidacao. +-- Split out of the monolithic liquidacao model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + liquidado_sc as ( + select + safe_cast(ano_emp as int64) as ano, + safe_cast(substring(trim(data_empenho), -7, 2) as int64) as mes, + safe_cast(null as date) as data, + 'SC' as sigla_uf, + safe_cast(id_municipio as string) as id_municipio, + safe_cast(codigo_orgao as string) as orgao, + safe_cast(null as string) as id_unidade_gestora, + safe_cast( + concat( + num_empenho, + ' ', + codigo_orgao, + ' ', + id_municipio, + ' ', + (right(cast(ano_emp as string), 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(num_empenho as string) as numero_empenho, + safe_cast(null as string) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(null as string) as numero, + safe_cast(null as string) as nome_responsavel, + safe_cast(null as string) as documento_responsavel, + safe_cast(null as bool) as indicador_restos_pagar, + round(safe_cast(0 as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_reforco, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(valor_liquidacao as float64), 2) as valor_final + from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_sc") }} + + ), + frequencia_sc as ( + select id_empenho_bd, count(id_empenho_bd) as frequencia_id + from liquidado_sc + group by 1 + order by 2 desc + + ), + liquidacao_sc as ( + select + l.ano, + l.mes, + l.data, + l.sigla_uf, + l.id_municipio, + l.orgao, + l.id_unidade_gestora, + ( + case + when frequencia_id > 1 + then (safe_cast(null as string)) + else l.id_empenho_bd + end + ) as id_empenho_bd, + l.id_empenho, + l.numero_empenho, + l.id_liquidacao_bd, + l.id_liquidacao, + l.numero, + l.nome_responsavel, + l.documento_responsavel, + l.indicador_restos_pagar, + l.valor_inicial, + l.valor_anulacao, + l.valor_ajuste, + l.valor_final + from liquidado_sc l + left join frequencia_sc f on l.id_empenho_bd = f.id_empenho_bd + + ) +select * +from liquidacao_sc diff --git a/models/world_wb_mides/states/world_wb_mides__liquidacao_sp.sql b/models/world_wb_mides/states/world_wb_mides__liquidacao_sp.sql new file mode 100644 index 0000000000..52b8706ca5 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__liquidacao_sp.sql @@ -0,0 +1,269 @@ +-- São Paulo (SP) contribution to world_wb_mides.liquidacao. +-- Split out of the monolithic liquidacao model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + liquidado_sp as ( + select + safe_cast(ano_exercicio as int64) as ano, + safe_cast(mes_referencia as int64) as mes, + safe_cast( + concat( + substring(dt_emissao_despesa, -4), + '-', + substring(dt_emissao_despesa, -7, 2), + '-', + substring(dt_emissao_despesa, 1, 2) + ) as date + ) as data, + 'SP' as sigla_uf, + safe_cast(id_municipio as string) as id_municipio, + safe_cast(codigo_orgao as string) as orgao, + safe_cast(null as string) as id_unidade_gestora, + safe_cast( + concat( + left(nr_empenho, length(nr_empenho) - 5), + ' ', + codigo_orgao, + ' ', + id_municipio, + ' ', + (right(ano_exercicio, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(nr_empenho as string) as numero_empenho, + safe_cast( + concat( + left(nr_empenho, length(nr_empenho) - 5), + ' ', + regexp_replace(identificador_despesa, '[^0-9]', ''), + ' ', + codigo_orgao, + ' ', + id_municipio, + ' ', + (right(ano_exercicio, 2)) + ) as string + ) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(null as string) as numero, + safe_cast(null as string) as nome_responsavel, + safe_cast(null as string) as documento_responsavel, + safe_cast(null as bool) as indicador_restos_pagar, + safe_cast(nr_empenho as string) as numero, + case + when ds_modalidade_lic = 'CONVITE' + then '1' + when ds_modalidade_lic = 'TOMADA DE PREÇOS' + then '2' + when ds_modalidade_lic = 'CONCORRÊNCIA' + then '3' + when ds_modalidade_lic = 'PREGÃO' + then '4' + when ds_modalidade_lic = 'Leilão' + then '7' + when ds_modalidade_lic = 'DISPENSA DE LICITAÇÃO' + then '8' + when ds_modalidade_lic = 'BEC-BOLSA ELETRÔNICA DE COMPRAS' + then '9' + when ds_modalidade_lic = 'INEXIGÍVEL' + then '10' + when ds_modalidade_lic = 'CONCURSO' + then '11' + when ds_modalidade_lic = 'RDC' + then '12' + when ds_modalidade_lic = 'OUTROS/NÃO APLICÁVEL' + then '99' + end as modalidade_licitacao, + safe_cast(lower(historico_despesa) as string) as descricao, + safe_cast(null as string) as modalidade, + safe_cast(funcao as string) as funcao, + safe_cast(subfuncao as string) as subfuncao, + safe_cast(cd_programa as string) as programa, + safe_cast(cd_acao as string) as acao, + safe_cast((left(ds_elemento, 8)) as string) as elemento_despesa, + safe_cast(replace(vl_despesa, ',', '.') as float64) as valor_inicial + from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_sp") }} e + left join + {{ set_datalake_project("world_wb_mides_staging.aux_municipio_sp") }} m + on m.ds_orgao = e.ds_orgao + left join + {{ set_datalake_project("world_wb_mides_staging.aux_funcao") }} + on ds_funcao_governo = upper(nome_funcao) + left join + {{ set_datalake_project("world_wb_mides_staging.aux_subfuncao") }} + on ds_subfuncao_governo = upper(nome_subfuncao) + where tp_despesa = 'Valor Liquidado' + + ), + dorgao as ( + select + id_empenho_bd, + case when (count(distinct orgao)) > 1 then 1 else 0 end as dorgao + from liquidado_sp + group by 1 + + ), + ddesc as ( + select + id_empenho_bd, + case + when (count(distinct ifnull(descricao, ''))) > 1 then 1 else 0 + end as ddesc + from liquidado_sp + group by 1 + + ), + dmod as ( + select + id_empenho_bd, + case + when (count(distinct modalidade_licitacao)) > 1 then 1 else 0 + end as dmod + from liquidado_sp + group by 1 + + ), + dfun as ( + select + id_empenho_bd, + case when (count(distinct funcao)) > 1 then 1 else 0 end as dfun + from liquidado_sp + group by 1 + + ), + dsubf as ( + select + id_empenho_bd, + case when (count(distinct subfuncao)) > 1 then 1 else 0 end as dsubf + from liquidado_sp + group by 1 + + ), + dprog as ( + select + id_empenho_bd, + case when (count(distinct programa)) > 1 then 1 else 0 end as dprog + from liquidado_sp + group by 1 + + ), + dacao as ( + select + id_empenho_bd, + case when (count(distinct acao)) > 1 then 1 else 0 end as dacao + from liquidado_sp + group by 1 + + ), + delem as ( + select + id_empenho_bd, + case when (count(distinct elemento_despesa)) > 1 then 1 else 0 end as delem + from liquidado_sp + group by 1 + + ), + dummies as ( + select o.id_empenho_bd, dorgao, dmod, ddesc, dfun, dsubf, dprog, dacao, delem + from dorgao o + full outer join dmod m on o.id_empenho_bd = m.id_empenho_bd + full outer join ddesc d on o.id_empenho_bd = d.id_empenho_bd + full outer join dfun f on o.id_empenho_bd = f.id_empenho_bd + full outer join dsubf s on o.id_empenho_bd = s.id_empenho_bd + full outer join dprog p on o.id_empenho_bd = p.id_empenho_bd + full outer join dacao a on o.id_empenho_bd = a.id_empenho_bd + full outer join delem e on o.id_empenho_bd = e.id_empenho_bd + + ), + liquidacao_sp as ( + select + min(ano) as ano, + min(mes) as mes, + min(data) as data, + sigla_uf, + id_municipio, + orgao, + id_unidade_gestora, + ( + case + when + ( + dorgao = 1 + or dmod = 1 + or dfun = 1 + or dsubf = 1 + or dprog = 1 + or dacao = 1 + or delem = 1 + ) + then (safe_cast(null as string)) + else l.id_empenho_bd + end + ) as id_empenho_bd, + id_empenho, + numero_empenho, + id_liquidacao_bd, + id_liquidacao, + safe_cast(null as string) as numero, + nome_responsavel, + documento_responsavel, + indicador_restos_pagar, + round(sum(valor_inicial), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(sum(valor_inicial), 2) as valor_final + from liquidado_sp l + left join dummies d on d.id_empenho_bd = l.id_empenho_bd + group by 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16 + + ), + liquidacao_municipio_sp as ( + select + (safe_cast(exercicio as int64)) as ano, + (safe_cast(extract(month from date(data_empenho)) as int64)) as mes, + safe_cast(data_empenho as date) as data, + 'SP' as sigla_uf, + '3550308' as id_municipio, + safe_cast(codigo_orgao as string) as orgao, + safe_cast(codigo_unidade as string) as id_unidade_gestora, + safe_cast( + concat( + nr_empenho, + ' ', + trim(codigo_orgao), + ' ', + trim(codigo_unidade), + ' ', + '3550308', + ' ', + (right(exercicio, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(id_empenho as string) as id_empenho, + safe_cast(nr_empenho as string) as numero_empenho, + safe_cast(null as string) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(null as string) as numero, + safe_cast(null as string) as nome_responsavel, + safe_cast(null as string) as documento_responsavel, + safe_cast(null as bool) as indicador_restos_pagar, + round(safe_cast(liquidado as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(liquidado as float64), 2) as valor_final + from + {{ + set_datalake_project( + "world_wb_mides_staging.raw_despesa_sp_municipio" + ) + }} + + ) +select * +from liquidacao_sp +union all +(select * from liquidacao_municipio_sp) diff --git a/models/world_wb_mides/states/world_wb_mides__pagamento_ce.sql b/models/world_wb_mides/states/world_wb_mides__pagamento_ce.sql new file mode 100644 index 0000000000..a2f2be24e4 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__pagamento_ce.sql @@ -0,0 +1,145 @@ +-- Ceará (CE) contribution to world_wb_mides.pagamento. +-- Split out of the monolithic pagamento model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + empenho_ce as ( + select + safe_cast( + concat( + numero_empenho, + ' ', + trim(codigo_orgao), + ' ', + trim(codigo_unidade), + ' ', + m.geoibgeid, + ' ', + (substring(data_emissao_empenho, 6, 2)), + ' ', + (substring(data_emissao_empenho, 3, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(nome_negociante as string) as nome_credor, + safe_cast( + replace ( + replace (numero_documento_negociante, '.', ''), '-', '' + ) as string + ) as documento_credor, + safe_cast(safe_cast(codigo_fonte_ as int64) as string) as fonte, + from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_ce") }} e + left join + {{ set_datalake_project("world_wb_mides_staging.aux_municipio_ce") }} m + on e.codigo_municipio = m.codigo_municipio + + ), + pago_ce as ( + select + (safe_cast(extract(year from date(data_nota_pagamento)) as int64)) as ano, + (safe_cast(extract(month from date(data_nota_pagamento)) as int64)) as mes, + safe_cast( + extract(date from timestamp(data_nota_pagamento)) as date + ) as data, + 'CE' as sigla_uf, + safe_cast(m.geoibgeid as string) as id_municipio, + safe_cast(p.codigo_orgao as string) orgao, + safe_cast(p.codigo_unidade as string) as id_unidade_gestora, + safe_cast( + concat( + p.numero_empenho, + ' ', + trim(p.codigo_orgao), + ' ', + trim(p.codigo_unidade), + ' ', + m.geoibgeid, + ' ', + (substring(p.data_emissao_empenho, 6, 2)), + ' ', + (substring(p.data_emissao_empenho, 3, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(p.numero_empenho as string) as numero_empenho, + safe_cast(null as string) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(null as string) as numero_liquidacao, + safe_cast( + concat( + p.numero_empenho, + ' ', + safe_cast(safe_cast(numero_nota_pagamento as int64) as string), + ' ', + trim(p.codigo_orgao), + ' ', + trim(p.codigo_unidade), + ' ', + m.geoibgeid, + ' ', + (substring(p.data_emissao_empenho, 6, 2)), + ' ', + (substring(p.data_emissao_empenho, 3, 2)) + ) as string + ) as id_pagamento_bd, + safe_cast(null as string) as id_pagamento, + safe_cast(numero_nota_pagamento as string) as numero, + safe_cast(null as bool) as indicador_restos_pagar, + round(safe_cast(valor_nota_pagamento as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(valor_nota_pagamento as float64), 2) as valor_final, + round(safe_cast(0 as float64), 2) as valor_liquido_recebido, + from {{ set_datalake_project("world_wb_mides_staging.raw_pagamento_ce") }} p + left join + {{ set_datalake_project("world_wb_mides_staging.aux_municipio_ce") }} m + on p.codigo_municipio = m.codigo_municipio + + ), + frequencia_ce as ( + select id_pagamento_bd, count(id_pagamento_bd) as frequencia_id + from pago_ce + group by 1 + + ), + pagamento_ce as ( + select + ano, + mes, + data, + sigla_uf, + id_municipio, + orgao, + id_unidade_gestora, + p.id_empenho_bd, + p.id_empenho, + p.numero_empenho, + id_liquidacao_bd, + id_liquidacao, + numero_liquidacao, + ( + case + when (frequencia_id > 1) + then (safe_cast(null as string)) + else p.id_pagamento_bd + end + ) as id_pagamento_bd, + id_pagamento, + numero, + nome_credor, + documento_credor, + indicador_restos_pagar, + fonte, + valor_inicial, + valor_anulacao, + valor_ajuste, + valor_final, + valor_liquido_recebido + from pago_ce p + left join frequencia_ce f on p.id_pagamento_bd = f.id_pagamento_bd + left join empenho_ce e on p.id_empenho_bd = e.id_empenho_bd + + ) +select * +from pagamento_ce diff --git a/models/world_wb_mides/states/world_wb_mides__pagamento_df.sql b/models/world_wb_mides/states/world_wb_mides__pagamento_df.sql new file mode 100644 index 0000000000..2b03c79646 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__pagamento_df.sql @@ -0,0 +1,107 @@ +-- Distrito Federal (DF) contribution to world_wb_mides.pagamento. +-- Split out of the monolithic pagamento model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + pagamento_df as ( + select + (safe_cast(exercicio as int64)) as ano, + safe_cast(substring(emissao, -7, 2) as int64) as mes, + safe_cast( + concat( + substring(emissao, -4), + '-', + substring(emissao, -7, 2), + '-', + substring(emissao, 1, 2) + ) as date + ) as data, + 'DF' as sigla_uf, + '5300108' as id_municipio, + safe_cast(codigo_ug as string) as orgao, + safe_cast(codigo_gestao as string) as id_unidade_gestora, + safe_cast( + concat( + right(nota_empenho, length(nota_empenho) - 6), + ' ', + codigo_ug, + ' ', + codigo_gestao, + ' ', + '5300108', + ' ', + (right(exercicio, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(nota_empenho as string) as numero_empenho, + case + when length(nota_lancamento) = 11 + then + safe_cast( + concat( + right(nota_lancamento, length(nota_lancamento) - 6), + ' ', + codigo_ug, + ' ', + codigo_gestao, + ' ', + '5300108', + ' ', + (right(exercicio, 2)) + ) as string + ) + end as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(nota_lancamento as string) as numero_liquidacao, + case + when length(numero_ordem_bancaria) = 11 + then + safe_cast( + concat( + right( + numero_ordem_bancaria, length(numero_ordem_bancaria) - 6 + ), + ' ', + codigo_ug, + ' ', + codigo_gestao, + ' ', + '5300108', + ' ', + (right(exercicio, 2)) + ) as string + ) + end as id_pagamento_bd, + safe_cast(null as string) as id_pagamento, + safe_cast(numero_ordem_bancaria as string) as numero, + safe_cast(credor as string) as nome_credor, + safe_cast(cnpj_cpf_credor as string) as documento_credor, + case + when ano_ordem_bancaria != ano_nota_empenho then true else false + end as indicador_restos_pagar, + safe_cast(null as string) as fonte, + round( + safe_cast(replace(valor_final_x, ',', '.') as float64), 2 + ) as valor_inicial, + round( + safe_cast(replace(valor_cancelado, ',', '.') as float64), 2 + ) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round( + safe_cast(replace(valor_final_x, ',', '.') as float64) + - safe_cast(replace(valor_cancelado, ',', '.') as float64), + 2 + ) as valor_final, + round( + safe_cast(replace(valor_final_x, ',', '.') as float64) + - safe_cast(replace(valor_cancelado, ',', '.') as float64), + 2 + ) as valor_liquido_recebido, + from {{ set_datalake_project("world_wb_mides_staging.raw_pagamento_df") }} + + ) +select * +from pagamento_df diff --git a/models/world_wb_mides/states/world_wb_mides__pagamento_mg.sql b/models/world_wb_mides/states/world_wb_mides__pagamento_mg.sql new file mode 100644 index 0000000000..93491ad95e --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__pagamento_mg.sql @@ -0,0 +1,118 @@ +-- Minas Gerais (MG) contribution to world_wb_mides.pagamento. +-- Split out of the monolithic pagamento model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + pagamento_mg as ( + select distinct + safe_cast(p.ano as int64) as ano, + safe_cast(p.mes as int64) as mes, + safe_cast(p.data as date) as data, + safe_cast(p.sigla_uf as string) as sigla_uf, + safe_cast(p.id_municipio as string) as id_municipio, + safe_cast(p.orgao as string) as orgao, + safe_cast(p.id_unidade_gestora as string) as id_unidade_gestora, + safe_cast( + case + when id_empenho != '-1' + then + concat( + id_empenho, + ' ', + p.orgao, + ' ', + p.id_municipio, + ' ', + (right(ano, 2)) + ) + when id_empenho = '-1' + then + concat( + id_empenho_origem, + ' ', + r.orgao, + ' ', + r.id_municipio, + ' ', + (right(num_ano_emp_origem, 2)) + ) + end as string + ) as id_empenho_bd, + safe_cast( + case + when p.id_empenho = '-1' + then replace (p.id_empenho, '-1', id_empenho_origem) + end as string + ) as id_empenho, + safe_cast(p.numero_empenho as string) as numero_empenho, + safe_cast( + case + when p.id_liquidacao != '-1' + then + concat( + p.id_liquidacao, + ' ', + p.orgao, + ' ', + p.id_municipio, + ' ', + (right(p.ano, 2)) + ) + when p.id_liquidacao = '-1' + then + concat( + ' ', r.orgao, ' ', r.id_municipio, ' ', (right(p.ano, 2)) + ) + end as string + ) as id_liquidacao_bd, + safe_cast( + case + when p.id_empenho = '-1' then replace (p.id_liquidacao, '-1', '') + end as string + ) as id_liquidacao, + safe_cast(p.numero_liquidacao as string) as numero_liquidacao, + safe_cast( + concat( + id_pagamento, + ' ', + p.orgao, + ' ', + p.id_municipio, + ' ', + (right(p.ano, 2)) + ) as string + ) as id_pagamento_bd, + safe_cast(id_pagamento as string) as id_pagamento, + safe_cast(p.numero_pagamento as string) as numero, + safe_cast(nome_credor as string) as nome_credor, + safe_cast( + replace(replace (documento_credor, '.', ''), '-', '') as string + ) as documento_credor, + safe_cast( + case when p.id_rsp != '-1' then 1 else 0 end as bool + ) as indicador_restos_pagar, + safe_cast(left(fonte, 3) as string) as fonte, + round(safe_cast(valor_pagamento_original as float64), 2) as valor_inicial, + round(ifnull(safe_cast(vlr_anu_fonte as float64), 0), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round( + safe_cast(valor_pagamento_original as float64) + - ifnull(safe_cast(vlr_anu_fonte as float64), 0), + 2 + ) as valor_final, + round( + safe_cast(valor_pagamento_original as float64) + - ifnull(safe_cast(vlr_anu_fonte as float64), 0) + - ifnull(safe_cast(vlr_ret_fonte as float64), 0), + 2 + ) as valor_liquido_recebido, + from {{ set_datalake_project("world_wb_mides_staging.raw_pagamento_mg") }} as p + left join + {{ set_datalake_project("world_wb_mides_staging.raw_rsp_mg") }} as r + on p.id_rsp = r.id_rsp + + ) +select * +from pagamento_mg diff --git a/models/world_wb_mides/states/world_wb_mides__pagamento_pb.sql b/models/world_wb_mides/states/world_wb_mides__pagamento_pb.sql new file mode 100644 index 0000000000..0cadf1cadf --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__pagamento_pb.sql @@ -0,0 +1,129 @@ +-- Paraíba (PB) contribution to world_wb_mides.pagamento. +-- Split out of the monolithic pagamento model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + pago_pb as ( + select + safe_cast(p.dt_ano as int64) as ano, + safe_cast(substring(trim(dt_pagamento), -7, 2) as int64) as mes, + safe_cast( + concat( + substring(trim(dt_pagamento), -4), + '-', + substring(trim(dt_pagamento), -7, 2), + '-', + substring(trim(dt_pagamento), 1, 2) + ) as date + ) as data, + m.sigla_uf, + safe_cast(m.id_municipio as string) as id_municipio, + safe_cast(null as string) as orgao, + safe_cast(p.cd_ugestora as string) as id_unidade_gestora, + safe_cast( + concat( + e.nu_empenho, + ' ', + e.cd_ugestora, + ' ', + m.id_municipio, + ' ', + (right(e.dt_ano, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(p.nu_empenho as string) as numero_empenho, + safe_cast(null as string) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(null as string) as numero_liquidacao, + safe_cast( + concat( + p.nu_empenho, + ' ', + (safe_cast(nu_parcela as int64)), + ' ', + p.cd_ugestora, + ' ', + id_municipio, + ' ', + (right(p.dt_ano, 2)) + ) as string + ) as id_pagamento_bd, + safe_cast(null as string) as id_pagamento, + safe_cast(nu_parcela as string) as numero, + safe_cast(no_credor as string) as nome_credor, + safe_cast( + replace (replace (cd_credor, '.', ''), '-', '') as string + ) as documento_credor, + safe_cast(null as bool) as indicador_restos_pagar, + safe_cast(tp_fonterecursos as string) as fonte, + round(safe_cast(vl_pagamento as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(vl_retencao as float64), 2) as valor_ajuste, + round(safe_cast(vl_pagamento as float64), 2) as valor_final, + round( + safe_cast(vl_pagamento as float64) - safe_cast(vl_retencao as float64), + 2 + ) as valor_liquido_recebido, + from {{ set_datalake_project("world_wb_mides_staging.raw_pagamento_pb") }} p + left join + {{ set_datalake_project("world_wb_mides_staging.raw_empenho_pb") }} e + on p.nu_empenho = e.nu_empenho + and p.cd_ugestora = e.cd_ugestora + and p.de_uorcamentaria = e.de_uorcamentaria + and p.dt_ano = e.dt_ano + left join + {{ set_datalake_project("world_wb_mides_staging.aux_municipio_pb") }} m + on safe_cast(e.cd_ugestora as string) + = safe_cast(m.id_unidade_gestora as string) + + ), + frequencia_pb as ( + select id_pagamento_bd, count(id_pagamento_bd) frequencia_id + from pago_pb + group by 1 + + ), + pagamento_pb as ( + select + ano, + mes, + data, + sigla_uf, + id_municipio, + orgao, + id_unidade_gestora, + case + when (frequencia_id > 1) + then (safe_cast(null as string)) + else p.id_empenho_bd + end as id_empenho_bd, + id_empenho, + numero_empenho, + id_liquidacao_bd, + id_liquidacao, + numero_liquidacao, + case + when (frequencia_id > 1) + then (safe_cast(null as string)) + else p.id_pagamento_bd + end as id_pagamento_bd, + id_pagamento, + numero, + nome_credor, + documento_credor, + indicador_restos_pagar, + fonte, + valor_inicial, + valor_anulacao, + valor_ajuste, + valor_final, + valor_liquido_recebido + from pago_pb p + left join frequencia_pb f on p.id_pagamento_bd = f.id_pagamento_bd + + ) +select * +from pagamento_pb diff --git a/models/world_wb_mides/states/world_wb_mides__pagamento_pe.sql b/models/world_wb_mides/states/world_wb_mides__pagamento_pe.sql new file mode 100644 index 0000000000..19bbc2e0cf --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__pagamento_pe.sql @@ -0,0 +1,52 @@ +-- Pernambuco (PE) contribution to world_wb_mides.pagamento. +-- Split out of the monolithic pagamento model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + pagamento_pe as ( + select + safe_cast(p.anoreferencia as int64) as ano, + (safe_cast(extract(month from date(data)) as int64)) as mes, + safe_cast(extract(date from timestamp(data)) as date) as data, + safe_cast(unidadefederativa as string) as sigla_uf, + safe_cast(codigoibge as string) as id_municipio, + safe_cast(null as string) orgao, + safe_cast(id_unidadegestora as string) as id_unidade_gestora, + safe_cast(null as string) as id_empenho_bd, + safe_cast(trim(idempenho) as string) as id_empenho, + safe_cast(p.numeroempenho as string) as numero_empenho, + safe_cast(null as string) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(null as string) as numero_liquidacao, + safe_cast(null as string) as id_pagamento_bd, + safe_cast(null as string) as id_pagamento, + safe_cast(null as string) as numero, + safe_cast(null as string) as nome_credor, + safe_cast(null as string) as documento_credor, + safe_cast(null as bool) as indicador_restos_pagar, + safe_cast(null as string) as fonte, + round(safe_cast(valor as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round( + ( + case + when (safe_cast((valor) as float64) < -1000000000000) + then null + else safe_cast((valor) as float64) + end + ), + 2 + ) as valor_final, + round(safe_cast(0 as float64), 2) as valor_liquido_recebido, + from {{ set_datalake_project("world_wb_mides_staging.raw_pagamento_pe") }} p + inner join + {{ set_datalake_project("world_wb_mides_staging.aux_municipio_pe") }} m + on safe_cast(p.id_unidade_gestora as string) + = safe_cast(m.id_unidadegestora as string) + + ) +select * +from pagamento_pe diff --git a/models/world_wb_mides/states/world_wb_mides__pagamento_pr.sql b/models/world_wb_mides/states/world_wb_mides__pagamento_pr.sql new file mode 100644 index 0000000000..1a928d3ead --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__pagamento_pr.sql @@ -0,0 +1,53 @@ +-- Paraná (PR) contribution to world_wb_mides.pagamento. +-- Split out of the monolithic pagamento model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + pagamento_pr as ( + select + safe_cast(nranopagamento as int64) as ano, + (safe_cast(extract(month from date(dtoperacao)) as int64)) as mes, + safe_cast(extract(date from timestamp(dtoperacao)) as date) as data, + sigla_uf, + id_municipio, + safe_cast(cdorgao as string) as orgao, + safe_cast(cdunidade as string) as id_unidade_gestora, + safe_cast( + concat(p.idempenho, ' ', m.id_municipio) as string + ) as id_empenho_bd, + safe_cast(p.idempenho as string) as id_empenho, + safe_cast(nrempenho as string) as numero_empenho, + safe_cast( + concat(p.idliquidacao, ' ', m.id_municipio) as string + ) as id_liquidacao_bd, + safe_cast(p.idliquidacao as string) as id_liquidacao, + safe_cast(null as string) as numero_liquidacao, + safe_cast( + concat(p.idpagamento, ' ', m.id_municipio) as string + ) as id_pagamento_bd, + safe_cast(idpagamento as string) as id_pagamento, + safe_cast(nrpagamento as string) as numero, + safe_cast(nmcredor as string) as nome_credor, + safe_cast( + regexp_replace(nrdoccredor, '[^0-9]', '') as string + ) as documento_credor, + safe_cast(null as bool) as indicador_restos_pagar, + safe_cast(cdfontereceita as string) as fonte, + round(safe_cast(vloperacao as float64), 2) as valor_inicial, + round(safe_cast(nranoliquidacao as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(p.cdibge as float64), 2) as valor_final, + round(safe_cast(0 as float64), 2) as valor_liquido_recebido, + from {{ set_datalake_project("world_wb_mides_staging.raw_pagamento_pr") }} p + left join + {{ set_datalake_project("world_wb_mides_staging.raw_empenho_pr") }} e + on p.idempenho = e.idempenho + left join + basedosdados.br_bd_diretorios_brasil.municipio m + on e.cdibge = id_municipio_6 + + ) +select * +from pagamento_pr diff --git a/models/world_wb_mides/states/world_wb_mides__pagamento_rj.sql b/models/world_wb_mides/states/world_wb_mides__pagamento_rj.sql new file mode 100644 index 0000000000..1e8c36faa9 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__pagamento_rj.sql @@ -0,0 +1,297 @@ +-- Rio de Janeiro (RJ) contribution to world_wb_mides.pagamento. +-- Split out of the monolithic pagamento model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + pago_municipio_rj_v1 as ( + select + safe_cast(exercicio_empenho as int64) as ano, + safe_cast(null as int64) as mes, + safe_cast(null as date) as data, + 'RJ' as sigla_uf, + '3304557' as id_municipio, + safe_cast(orgao_programa_trabalho as string) as orgao, + safe_cast(unidade_programa_trabalho as string) as id_unidade_gestora, + safe_cast( + concat( + nr_empenho, + ' ', + trim(orgao_programa_trabalho), + ' ', + trim(unidade_programa_trabalho), + ' ', + '3304557', + ' ', + (right(exercicio_empenho, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(nr_empenho as string) as numero_empenho, + safe_cast(null as string) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(null as string) as numero_liquidacao, + safe_cast(null as string) as id_pagamento_bd, + safe_cast(null as string) as id_pagamento, + safe_cast(null as string) as numero, + safe_cast(favorecido as string) as nome_credor, + safe_cast(codigo_favorecido as string) as documento_credor, + safe_cast(null as bool) as indicador_restos_pagar, + safe_cast(fonte_recursos as string) as fonte, + round(safe_cast(0 as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(valor_pago as float64), 2) as valor_final, + round(safe_cast(valor_pago as float64), 2) as valor_liquido_recebido + from + {{ + set_datalake_project( + "world_wb_mides_staging.raw_despesa_rj_municipio" + ) + }} + + ), + frequencia_rj_v1 as ( + select id_empenho_bd, count(id_empenho_bd) as frequencia_id + from pago_municipio_rj_v1 + group by 1 + order by 2 desc + + ), + pagamento_municipio_rj_v1 as ( + select + p.ano, + p.mes, + p.data, + p.sigla_uf, + p.id_municipio, + p.orgao, + p.id_unidade_gestora, + ( + case + when frequencia_id > 1 + then (safe_cast(null as string)) + else p.id_empenho_bd + end + ) as id_empenho_bd, + p.id_empenho, + p.numero_empenho, + p.id_liquidacao_bd, + p.id_liquidacao, + p.numero_liquidacao, + p.id_pagamento_bd, + p.id_pagamento, + p.numero, + p.nome_credor, + p.documento_credor, + p.indicador_restos_pagar, + p.fonte, + p.valor_inicial, + p.valor_anulacao, + p.valor_ajuste, + p.valor_final, + p.valor_liquido_recebido + from pago_municipio_rj_v1 p + left join frequencia_rj_v1 f on p.id_empenho_bd = f.id_empenho_bd + + ), + pago_municipio_rj_v2 as ( + select + (safe_cast(exercicio as int64)) as ano, + (safe_cast(extract(month from date(data)) as int64)) as mes, + safe_cast(data as date) as data, + 'RJ' as sigla_uf, + '3304557' as id_municipio, + safe_cast(ug as string) as orgao, + safe_cast(uo as string) as id_unidade_gestora, + safe_cast( + concat( + left(empenhoexercicio, length(empenhoexercicio) - 5), + ' ', + trim(uo), + ' ', + trim(ug), + ' ', + '3304557', + ' ', + (right(exercicio, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(empenhoexercicio as string) as numero_empenho, + safe_cast( + concat( + liquidacao, + ' ', + left(empenhoexercicio, length(empenhoexercicio) - 5), + ' ', + trim(uo), + ' ', + trim(ug), + ' ', + '3304557', + ' ', + (right(exercicio, 2)) + ) as string + ) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(liquidacao as string) as numero_liquidacao, + safe_cast( + concat( + pagamento, + ' ', + left(empenhoexercicio, length(empenhoexercicio) - 5), + ' ', + trim(uo), + ' ', + trim(ug), + ' ', + '3304557', + ' ', + (right(exercicio, 2)) + ) as string + ) as id_pagamento_bd, + safe_cast(null as string) as id_pagamento, + safe_cast(pagamento as string) as numero, + safe_cast(nomecredor as string) as nome_credor, + safe_cast(credor as string) as documento_credor, + safe_cast(null as bool) as indicador_restos_pagar, + safe_cast(fonterecursos as string) as fonte, + round(safe_cast(valor as float64), 2) as valor_inicial, + from + {{ + set_datalake_project( + "world_wb_mides_staging.raw_despesa_ato_rj_municipio" + ) + }} + where tipoato = 'PAGAMENTO' + + ), + anulacao_municipio_rj_v2 as ( + select + safe_cast(tipoato as string) as tipoato, + safe_cast( + concat( + left(empenhoexercicio, length(empenhoexercicio) - 5), + ' ', + trim(uo), + ' ', + trim(ug), + ' ', + '3304557', + ' ', + (right(exercicio, 2)) + ) as string + ) as id_empenho_bd, + sum(safe_cast(valor as float64)) as valor_anulacao, + from + {{ + set_datalake_project( + "world_wb_mides_staging.raw_despesa_ato_rj_municipio" + ) + }} + where + tipoato in ( + 'CANCEL.PAGAMENTO RET.DOTAÇÃO', + 'CANCEL.PAGAMENTO RET.EMPENHO', + 'CANCELAMENTO DE PAGAMENTO DE RPP', + 'CANCELAMENTO DE PAGAMENTO DE RPN', + 'Cancelamento de RPP' + ) + group by 1, 2 + + ), + frequencia_rj_v2 as ( + select id_empenho_bd, count(1) as frequencia + from anulacao_municipio_rj_v2 + group by 1 + + ), + pagamento_municipio_rj_v2 as ( + select + p.ano, + p.mes, + p.data, + p.sigla_uf, + p.id_municipio, + p.orgao, + p.id_unidade_gestora, + p.id_empenho_bd, + p.id_empenho, + p.numero_empenho, + p.id_liquidacao_bd, + p.id_liquidacao, + p.numero_liquidacao, + p.id_empenho_bd, + p.id_empenho, + p.numero, + p.nome_credor, + p.documento_credor, + case + when tipoato = 'CANCELAMENTO DE PAGAMENTO DE RPP' + then true + when tipoato = 'CANCELAMENTO DE PAGAMENTO DE RPN' + then true + when tipoato = 'Cancelamento de RPP' + then true + else false + end as indicador_restos_pagar, + p.fonte, + round(safe_cast(p.valor_inicial as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(p.valor_inicial as float64), 2) as valor_final, + round(safe_cast(p.valor_inicial as float64), 2) as valor_liquido_recebido + from pago_municipio_rj_v2 p + left join anulacao_municipio_rj_v2 a on p.id_empenho_bd = a.id_empenho_bd + left join frequencia_rj_v2 f on p.id_empenho_bd = f.id_empenho_bd + + ), + pagamento_rj as ( + select + (safe_cast(ano as int64)) as ano, + (safe_cast(extract(month from date(data)) as int64)) as mes, + safe_cast(data as date) as data, + 'RJ' as sigla_uf, + safe_cast(id_municipio as string) as id_municipio, + safe_cast(id_orgao as string) as orgao, + safe_cast(unidade_administrativa as string) as id_unidade_gestora, + safe_cast( + concat( + numero_empenho, + ' ', + id_orgao, + ' ', + id_municipio, + ' ', + (right(ano, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(numero_empenho as string) as numero_empenho, + safe_cast(null as string) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(null as string) as numero_liquidacao, + safe_cast(null as string) as id_pagamento_bd, + safe_cast(null as string) as id_pagamento, + safe_cast(null as string) as numero, + safe_cast(credor as string) as nome_credor, + safe_cast(null as string) as documento_credor, + safe_cast(null as bool) as indicador_restos_pagar, + safe_cast(fonte as string) as fonte, + round(safe_cast(valor as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(valor as float64), 2) as valor_final, + round(safe_cast(valor as float64), 2) as valor_liquido_recebido, + from {{ set_datalake_project("world_wb_mides_staging.raw_pagamento_rj") }} + where numero_empenho is not null + + ) +select * +from pagamento_municipio_rj_v1 +union all +(select * from pagamento_municipio_rj_v2) +union all +(select * from pagamento_rj) diff --git a/models/world_wb_mides/states/world_wb_mides__pagamento_rs.sql b/models/world_wb_mides/states/world_wb_mides__pagamento_rs.sql new file mode 100644 index 0000000000..2fec1b6a44 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__pagamento_rs.sql @@ -0,0 +1,250 @@ +-- Rio Grande do Sul (RS) contribution to world_wb_mides.pagamento. +-- Split out of the monolithic pagamento model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + pago_rs as ( + select + min(ano_recebimento) as ano_recebimento, + safe_cast(ano_operacao as int64) as ano, + safe_cast(extract(month from date(dt_operacao)) as int64) as mes, + safe_cast( + concat( + substring(dt_operacao, 1, 4), + '-', + substring(dt_operacao, 6, 2), + '-', + substring(dt_operacao, 9, 2) + ) as date + ) as data, + m.sigla_uf as sigla_uf, + safe_cast(a.id_municipio as string) as id_municipio, + safe_cast(c.cd_orgao as string) as orgao, + safe_cast(cd_orgao_orcamentario as string) as id_unidade_gestora, + safe_cast( + concat( + nr_empenho, + ' ', + c.cd_orgao, + ' ', + m.id_municipio, + ' ', + (right(ano_empenho, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(nr_empenho as string) as numero_empenho, + safe_cast( + concat( + nr_empenho, + ' ', + nr_liquidacao, + ' ', + c.cd_orgao, + ' ', + m.id_municipio, + ' ', + (right(ano_empenho, 2)) + ) as string + ) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(nr_liquidacao as string) as numero_liquidacao, + safe_cast( + concat( + nr_empenho, + ' ', + nr_liquidacao, + ' ', + nr_pagamento, + ' ', + c.cd_orgao, + ' ', + m.id_municipio, + ' ', + (right(ano_empenho, 2)) + ) as string + ) as id_pagamento_bd, + safe_cast(null as string) as id_pagamento, + safe_cast(nr_pagamento as string) as numero, + safe_cast(nm_credor as string) as nome_credor, + safe_cast(cnpj_cpf as string) as documento_credor, + safe_cast(null as bool) as indicador_restos_pagar, + safe_cast(null as string) as fonte, + safe_cast(vl_pagamento as float64) as valor_inicial + from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} as c + left join + {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} as a + on c.cd_orgao = a.cd_orgao + left join + `basedosdados.br_bd_diretorios_brasil.municipio` m + on m.id_municipio = a.id_municipio + where tipo_operacao = 'P' and (safe_cast(vl_pagamento as float64) >= 0) + group by + 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22 + + ), + estorno_rs as ( + select + safe_cast( + concat( + nr_empenho, + ' ', + c.cd_orgao, + ' ', + m.id_municipio, + ' ', + (right(ano_empenho, 2)) + ) as string + ) as id_empenho_bd, + -1 * sum(safe_cast(vl_pagamento as float64)) as valor_anulacao + from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} as c + left join + {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} as a + on c.cd_orgao = a.cd_orgao + left join + `basedosdados.br_bd_diretorios_brasil.municipio` m + on m.id_municipio = a.id_municipio + where tipo_operacao = 'P' and (safe_cast(vl_pagamento as float64) < 0) + group by 1 + + ), + frequencia_rs as ( + select id_empenho_bd, count(id_empenho_bd) as frequencia_id + from pago_rs + group by 1 + + ), + pagamento1_rs as ( + select + ano, + mes, + data, + sigla_uf, + id_municipio, + orgao, + id_unidade_gestora, + p.id_empenho_bd, + id_empenho, + numero_empenho, + p.id_liquidacao_bd, + id_liquidacao, + numero_liquidacao, + id_pagamento_bd, + id_pagamento, + numero, + nome_credor, + ifnull(documento_credor, '99999999999') as documento_credor, + indicador_restos_pagar, + fonte, + round(sum(valor_inicial), 2) as valor_inicial, + round(sum(valor_anulacao / frequencia_id), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round( + sum(valor_inicial - ifnull((valor_anulacao / frequencia_id), 0)), 2 + ) as valor_final, + round(safe_cast(0 as float64), 2) as valor_liquido_recebido + from pago_rs p + left join estorno_rs e on p.id_empenho_bd = e.id_empenho_bd + left join frequencia_rs f on p.id_empenho_bd = f.id_empenho_bd + group by 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 + + ), + ddata_rs as ( + select + id_pagamento_bd, + case when (count(distinct data)) > 1 then 1 else 0 end as ddata + from pagamento1_rs + group by 1 + + ), + dorgao_rs as ( + select + id_pagamento_bd, + case when (count(distinct orgao)) > 1 then 1 else 0 end as dorgao + from pagamento1_rs + group by 1 + + ), + dugest_rs as ( + select + id_pagamento_bd, + case + when (count(distinct id_unidade_gestora)) > 1 then 1 else 0 + end as dugest + from pagamento1_rs + group by 1 + + ), + credor_rs as ( + select + id_pagamento_bd, + case when (count(distinct nome_credor)) > 1 then 1 else 0 end as dcredor + from pagamento1_rs + group by 1 + + ), + dcredor_rs as ( + select + id_pagamento_bd, + case + when (count(distinct documento_credor)) > 1 then 1 else 0 + end as ddocumento + from pagamento1_rs + group by 1 + + ), + dummies as ( + select d.id_pagamento_bd, ddata, dorgao, dugest, dcredor, ddocumento + from ddata_rs d + left join credor_rs c on d.id_pagamento_bd = c.id_pagamento_bd + left join dcredor_rs dc on d.id_pagamento_bd = dc.id_pagamento_bd + left join dorgao_rs o on d.id_pagamento_bd = o.id_pagamento_bd + left join dugest_rs u on d.id_pagamento_bd = u.id_pagamento_bd + + ), + pagamento_rs as ( + select + ano, + mes, + data, + sigla_uf, + id_municipio, + orgao, + id_unidade_gestora, + id_empenho_bd, + id_empenho, + numero_empenho, + id_liquidacao_bd, + id_liquidacao, + numero_liquidacao, + case + when + ddata = 1 + or dorgao = 1 + or dugest = 1 + or dcredor = 1 + or ddocumento = 1 + or (numero_liquidacao = '0' and valor_final = 0) + or (numero = '0' and valor_final = 0) + then (safe_cast(null as string)) + else p.id_pagamento_bd + end as id_pagamento_bd, + id_pagamento, + numero, + nome_credor, + documento_credor, + indicador_restos_pagar, + fonte, + valor_inicial, + valor_anulacao, + valor_ajuste, + valor_final, + valor_liquido_recebido + from pagamento1_rs p + left join dummies d on p.id_pagamento_bd = d.id_pagamento_bd + + ) +select * +from pagamento_rs diff --git a/models/world_wb_mides/states/world_wb_mides__pagamento_sc.sql b/models/world_wb_mides/states/world_wb_mides__pagamento_sc.sql new file mode 100644 index 0000000000..88ed4c141e --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__pagamento_sc.sql @@ -0,0 +1,93 @@ +-- Santa Catarina (SC) contribution to world_wb_mides.pagamento. +-- Split out of the monolithic pagamento model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + pago_sc as ( + select + safe_cast(ano_emp as int64) as ano, + safe_cast(substring(trim(data_empenho), -7, 2) as int64) as mes, + safe_cast(null as date) as data, + 'SC' as sigla_uf, + safe_cast(id_municipio as string) as id_municipio, + safe_cast(codigo_orgao as string) as orgao, + safe_cast(null as string) as id_unidade_gestora, + safe_cast( + concat( + num_empenho, + ' ', + codigo_orgao, + ' ', + id_municipio, + ' ', + (right(cast(ano_emp as string), 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(num_empenho as string) as numero_empenho, + safe_cast(null as string) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(null as string) as numero_liquidacao, + safe_cast(null as string) as id_pagamento_bd, + safe_cast(null as string) as id_pagamento, + safe_cast(null as string) as numero, + safe_cast(nome_credor as string) as nome_credor, + safe_cast(cpf_cnpj as string) as documento_credor, + safe_cast(null as bool) as indicador_restos_pagar, + safe_cast(right(especificacao_fonte_recurso, 2) as string) as fonte, + round(safe_cast(0 as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(valor_pagamento as float64), 2) as valor_final, + round(safe_cast(valor_pagamento as float64), 2) as valor_liquido_recebido + from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_sc") }} + + ), + frequencia_sc as ( + select id_empenho_bd, count(id_empenho_bd) as frequencia_id + from pago_sc + group by 1 + order by 2 desc + + ), + pagamento_sc as ( + select + p.ano, + p.mes, + p.data, + p.sigla_uf, + p.id_municipio, + p.orgao, + p.id_unidade_gestora, + ( + case + when frequencia_id > 1 + then (safe_cast(null as string)) + else p.id_empenho_bd + end + ) as id_empenho_bd, + p.id_empenho, + p.numero_empenho, + p.id_liquidacao_bd, + p.id_liquidacao, + p.numero_liquidacao, + p.id_pagamento_bd, + p.id_pagamento, + p.numero, + p.nome_credor, + p.documento_credor, + p.indicador_restos_pagar, + p.fonte, + p.valor_inicial, + p.valor_anulacao, + p.valor_ajuste, + p.valor_final, + p.valor_liquido_recebido + from pago_sc p + left join frequencia_sc f on p.id_empenho_bd = f.id_empenho_bd + + ) +select * +from pagamento_sc diff --git a/models/world_wb_mides/states/world_wb_mides__pagamento_sp.sql b/models/world_wb_mides/states/world_wb_mides__pagamento_sp.sql new file mode 100644 index 0000000000..ee61831626 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__pagamento_sp.sql @@ -0,0 +1,367 @@ +-- São Paulo (SP) contribution to world_wb_mides.pagamento. +-- Split out of the monolithic pagamento model. The SQL is unchanged: the CTE +-- bodies are byte-identical and the union order is preserved, so this model +-- emits exactly the rows and column positions it did before the split. +-- Column names are positional -- the parent model applies the canonical list. +-- Materialisation is set in dbt_project.yml (models/world_wb_mides/states). +with + pago_sp as ( + select + safe_cast(ano_exercicio as int64) as ano, + safe_cast(mes_referencia as int64) as mes, + safe_cast( + concat( + substring(dt_emissao_despesa, -4), + '-', + substring(dt_emissao_despesa, -7, 2), + '-', + substring(dt_emissao_despesa, 1, 2) + ) as date + ) as data, + sigla_uf, + safe_cast(id_municipio as string) as id_municipio, + safe_cast(codigo_orgao as string) as orgao, + safe_cast(null as string) as id_unidade_gestora, + safe_cast( + concat( + left(nr_empenho, length(nr_empenho) - 5), + ' ', + codigo_orgao, + ' ', + id_municipio, + ' ', + (right(ano_exercicio, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + safe_cast(nr_empenho as string) as numero_empenho, + safe_cast( + concat( + left(nr_empenho, length(nr_empenho) - 5), + ' ', + regexp_replace(identificador_despesa, '[^0-9]', ''), + ' ', + codigo_orgao, + ' ', + id_municipio, + ' ', + (right(ano_exercicio, 2)) + ) as string + ) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast( + concat( + left(nr_empenho, length(nr_empenho) - 5), + ' ', + regexp_replace(identificador_despesa, '[^0-9]', ''), + ' ', + codigo_orgao, + ' ', + id_municipio, + ' ', + (right(ano_exercicio, 2)) + ) as string + ) as id_pagamento_bd, + safe_cast(null as string) as numero_liquidacao, + safe_cast(null as string) as id_pagamento, + safe_cast(null as string) as numero, + safe_cast(ds_despesa as string) as nome_credor, + safe_cast( + regexp_replace(identificador_despesa, '[^0-9]', '') as string + ) as documento_credor, + safe_cast(null as bool) as indicador_restos_pagar, + safe_cast(null as string) as fonte, + case + when ds_modalidade_lic = 'CONVITE' + then '1' + when ds_modalidade_lic = 'TOMADA DE PREÇOS' + then '2' + when ds_modalidade_lic = 'CONCORRÊNCIA' + then '3' + when ds_modalidade_lic = 'PREGÃO' + then '4' + when ds_modalidade_lic = 'Leilão' + then '7' + when ds_modalidade_lic = 'DISPENSA DE LICITAÇÃO' + then '8' + when ds_modalidade_lic = 'BEC-BOLSA ELETRÔNICA DE COMPRAS' + then '9' + when ds_modalidade_lic = 'INEXIGÍVEL' + then '10' + when ds_modalidade_lic = 'CONCURSO' + then '11' + when ds_modalidade_lic = 'RDC' + then '12' + when ds_modalidade_lic = 'OUTROS/NÃO APLICÁVEL' + then '99' + end as modalidade_licitacao, + safe_cast(lower(historico_despesa) as string) as descricao, + safe_cast(null as string) as modalidade, + safe_cast(funcao as string) as funcao, + safe_cast(subfuncao as string) as subfuncao, + safe_cast(cd_programa as string) as programa, + safe_cast(cd_acao as string) as acao, + safe_cast((left(ds_elemento, 8)) as string) as elemento_despesa, + safe_cast(replace(vl_despesa, ',', '.') as float64) as valor_inicial + from {{ set_datalake_project("world_wb_mides_staging.raw_despesa_sp") }} e + left join + {{ set_datalake_project("world_wb_mides_staging.aux_municipio_sp") }} m + on m.ds_orgao = e.ds_orgao + left join + {{ set_datalake_project("world_wb_mides_staging.aux_funcao") }} + on ds_funcao_governo = upper(nome_funcao) + left join + {{ set_datalake_project("world_wb_mides_staging.aux_subfuncao") }} + on ds_subfuncao_governo = upper(nome_subfuncao) + where tp_despesa = 'Valor Pago' + + ), + dorgao as ( + select + id_empenho_bd, + case when (count(distinct orgao)) > 1 then 1 else 0 end as dorgao + from pago_sp + group by 1 + + ), + ddesc as ( + select + id_empenho_bd, + case + when (count(distinct ifnull(descricao, ''))) > 1 then 1 else 0 + end as ddesc + from pago_sp + group by 1 + + ), + dmod as ( + select + id_empenho_bd, + case + when (count(distinct modalidade_licitacao)) > 1 then 1 else 0 + end as dmod + from pago_sp + group by 1 + + ), + dfun as ( + select + id_empenho_bd, + case when (count(distinct funcao)) > 1 then 1 else 0 end as dfun + from pago_sp + group by 1 + + ), + dsubf as ( + select + id_empenho_bd, + case when (count(distinct subfuncao)) > 1 then 1 else 0 end as dsubf + from pago_sp + group by 1 + + ), + dprog as ( + select + id_empenho_bd, + case when (count(distinct programa)) > 1 then 1 else 0 end as dprog + from pago_sp + group by 1 + + ), + dacao as ( + select + id_empenho_bd, + case when (count(distinct acao)) > 1 then 1 else 0 end as dacao + from pago_sp + group by 1 + + ), + delem as ( + select + id_empenho_bd, + case when (count(distinct elemento_despesa)) > 1 then 1 else 0 end as delem + from pago_sp + group by 1 + + ), + dcredor as ( + select + id_empenho_bd, + case when (count(distinct nome_credor)) > 1 then 1 else 0 end as dcredor + from pago_sp + group by 1 + + ), + ddocumento as ( + select + id_empenho_bd, + case + when (count(distinct documento_credor)) > 1 then 1 else 0 + end as ddocumento + from pago_sp + group by 1 + + ), + dummies_sp as ( + select + o.id_empenho_bd, + dorgao, + dmod, + ddesc, + dfun, + dsubf, + dprog, + dacao, + delem, + dcredor, + ddocumento + from dorgao o + full outer join dmod m on o.id_empenho_bd = m.id_empenho_bd + full outer join ddesc d on o.id_empenho_bd = d.id_empenho_bd + full outer join dfun f on o.id_empenho_bd = f.id_empenho_bd + full outer join dsubf s on o.id_empenho_bd = s.id_empenho_bd + full outer join dprog p on o.id_empenho_bd = p.id_empenho_bd + full outer join dacao a on o.id_empenho_bd = a.id_empenho_bd + full outer join delem e on o.id_empenho_bd = e.id_empenho_bd + full outer join dcredor c on o.id_empenho_bd = c.id_empenho_bd + full outer join ddocumento dc on o.id_empenho_bd = dc.id_empenho_bd + + ), + frequencia_pg_sp as ( + select id_pagamento_bd, count(id_pagamento_bd) frequencia_id + from pago_sp + group by 1 + + ), + pagamento_sp as ( + select + min(ano) as ano, + min(mes) as mes, + min(data) as data, + sigla_uf, + id_municipio, + orgao, + id_unidade_gestora, + ( + case + when + ( + dorgao = 1 + or dmod = 1 + or dfun = 1 + or dsubf = 1 + or dprog = 1 + or dacao = 1 + or delem = 1 + ) + then (safe_cast(null as string)) + else p.id_empenho_bd + end + ) as id_empenho_bd, + id_empenho, + numero_empenho, + ( + case + when + ( + dorgao = 1 + or dmod = 1 + or dfun = 1 + or dsubf = 1 + or dprog = 1 + or dacao = 1 + or delem = 1 + ) + then (safe_cast(null as string)) + else p.id_liquidacao_bd + end + ) as id_liquidacao_bd, + id_liquidacao, + numero_liquidacao, + ( + case + when + ( + dorgao = 1 + or dmod = 1 + or dfun = 1 + or dsubf = 1 + or dprog = 1 + or dacao = 1 + or delem = 1 + or dcredor = 1 + or ddocumento = 1 + ) + or frequencia_id > 1 + then (safe_cast(null as string)) + else p.id_pagamento_bd + end + ) as id_pagamento_bd, + id_pagamento, + numero, + nome_credor, + documento_credor, + indicador_restos_pagar, + fonte, + round(safe_cast(sum(valor_inicial) as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(sum(valor_inicial) as float64), 2) as valor_final, + round(safe_cast(0 as float64), 2) as valor_liquido_recebido + from pago_sp p + left join dummies_sp d on d.id_empenho_bd = p.id_empenho_bd + left join frequencia_pg_sp f on f.id_pagamento_bd = p.id_pagamento_bd + group by 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 + + ), + pagamento_municipio_sp as ( + select + (safe_cast(exercicio as int64)) as ano, + (safe_cast(extract(month from date(data_empenho)) as int64)) as mes, + safe_cast(data_empenho as date) as data, + 'SP' as sigla_uf, + '3550308' as id_municipio, + safe_cast(codigo_orgao as string) as orgao, + safe_cast(codigo_unidade as string) as id_unidade_gestora, + safe_cast( + concat( + nr_empenho, + ' ', + trim(codigo_orgao), + ' ', + trim(codigo_unidade), + ' ', + '3550308', + ' ', + (right(exercicio, 2)) + ) as string + ) as id_empenho_bd, + safe_cast(id_empenho as string) as id_empenho, + safe_cast(nr_empenho as string) as numero_empenho, + safe_cast(null as string) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + safe_cast(null as string) as numero_liquidacao, + safe_cast(null as string) as id_pagamento_bd, + safe_cast(null as string) as id_pagamento, + safe_cast(null as string) as numero, + safe_cast(razao_social as string) as nome_credor, + safe_cast(cpf_cnpj as string) as documento_credor, + safe_cast(null as bool) as indicador_restos_pagar, + safe_cast(codigo_fonte_recurso as string) as fonte, + round(safe_cast(pago as float64), 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(safe_cast(pago as float64), 2) as valor_final, + round(safe_cast(pago as float64), 2) as valor_liquido_recebido, + from + {{ + set_datalake_project( + "world_wb_mides_staging.raw_despesa_sp_municipio" + ) + }} + + ) +select * +from pagamento_sp +union all +(select * from pagamento_municipio_sp) diff --git a/models/world_wb_mides/world_wb_mides__empenho.sql b/models/world_wb_mides/world_wb_mides__empenho.sql index 21fec9a207..c9054ad05a 100644 --- a/models/world_wb_mides/world_wb_mides__empenho.sql +++ b/models/world_wb_mides/world_wb_mides__empenho.sql @@ -40,2201 +40,33 @@ select valor_final from ( - with - empenhado_ce as ( - select - ( - safe_cast( - extract(year from date(data_emissao_empenho)) as int64 - ) - ) as ano, - ( - safe_cast( - extract(month from date(data_emissao_empenho)) as int64 - ) - ) as mes, - safe_cast( - extract(date from timestamp(data_emissao_empenho)) as date - ) as data, - 'CE' as sigla_uf, - safe_cast(geoibgeid as string) as id_municipio, - safe_cast(codigo_orgao as string) as orgao, - safe_cast(trim(codigo_unidade) as string) as id_unidade_gestora, - safe_cast(null as string) as id_licitacao_bd, - safe_cast(numero_licitacao as string) as id_licitacao, - case - when tipo_processo_licitatorio = 'N' - then '98' - when tipo_processo_licitatorio = 'R' - then '2' - when tipo_processo_licitatorio = 'D' - then '8' - when tipo_processo_licitatorio = 'I' - then '10' - when tipo_processo_licitatorio = 'R' - then '29' - end as modalidade_licitacao, - safe_cast( - concat( - numero_empenho, - ' ', - trim(codigo_orgao), - ' ', - trim(codigo_unidade), - ' ', - geoibgeid, - ' ', - (substring(data_emissao_empenho, 6, 2)), - ' ', - (substring(data_emissao_empenho, 3, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(numero_empenho as string) as numero, - safe_cast(lower(descricao_empenho) as string) as descricao, - safe_cast(modalidade_empenho as string) as modalidade, - safe_cast(safe_cast(codigo_funcao as int64) as string) as funcao, - safe_cast( - safe_cast(codigo_subfuncao as int64) as string - ) as subfuncao, - safe_cast( - safe_cast(codigo_programa as int64) as string - ) as programa, - safe_cast( - safe_cast(codigo_projeto_atividade as int64) as string - ) as acao, - safe_cast( - safe_cast(codigo_elemento_despesa as int64) as string - ) as modalidade_despesa, - round(safe_cast(valor_empenhado as float64), 2) as valor_inicial, - from - {{ set_datalake_project("world_wb_mides_staging.raw_empenho_ce") }} e - ), - anulacao_ce as ( - select - safe_cast( - concat( - numero_empenho, - ' ', - trim(codigo_orgao), - ' ', - trim(codigo_unidade), - ' ', - geoibgeid, - ' ', - (substring(data_emissao_empenho, 6, 2)), - ' ', - (substring(data_emissao_empenho, 3, 2)) - ) as string - ) as id_empenho_bd, - round( - sum(safe_cast(valor_anulacao as float64)), 2 - ) as valor_anulacao - from - {{ set_datalake_project("world_wb_mides_staging.raw_anulacao_ce") }} - group by 1 - ), - frequencia_ce as ( - select id_empenho_bd, count(id_empenho_bd) as frequencia_id - from empenhado_ce - group by 1 - order by 2 desc - ), - empenho_ce as ( - select - e.ano, - e.mes, - e.data, - e.sigla_uf, - e.id_municipio, - e.orgao, - e.id_unidade_gestora, - e.id_licitacao_bd, - e.id_licitacao, - e.modalidade_licitacao, - ( - case - when frequencia_id > 1 - then (safe_cast(null as string)) - else e.id_empenho_bd - end - ) as id_empenho_bd, - e.id_empenho, - e.numero, - e.descricao, - e.modalidade, - e.funcao, - e.subfuncao, - e.programa, - e.acao, - e.modalidade_despesa, - round(e.valor_inicial, 2), - round(safe_cast(0 as float64), 2) as valor_reforco, - round(a.valor_anulacao, 2), - round(safe_cast(0 as float64), 2) as valor_ajuste, - round( - ifnull(e.valor_inicial, 0) - ifnull(a.valor_anulacao, 0), 2 - ) as valor_final - from empenhado_ce e - left join frequencia_ce f on e.id_empenho_bd = f.id_empenho_bd - full outer join anulacao_ce a on a.id_empenho_bd = e.id_empenho_bd - ), - empenhado_mg as ( - select - safe_cast(ano as int64) as ano, - safe_cast(mes as int64) as mes, - safe_cast(data as date) as data, - 'MG' as sigla_uf, - safe_cast(id_municipio as string) as id_municipio, - safe_cast(trim(orgao) as string) as orgao, - safe_cast(id_unidade_gestora as string) as id_unidade_gestora, - safe_cast(null as string) as id_licitacao_bd, - safe_cast(id_licitacao as string) as id_licitacao, - safe_cast(null as string) as modalidade_licitacao, - safe_cast( - concat( - id_empenho, - ' ', - orgao, - ' ', - id_municipio, - ' ', - (right(ano, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(id_empenho as string) as id_empenho, - safe_cast(numero_empenho as string) as numero, - safe_cast(lower(descricao) as string) as descricao, - safe_cast(substring(dsc_modalidade, 5, 1) as string) as modalidade, - safe_cast(cast(left(dsc_funcao, 2) as int64) as string) as funcao, - safe_cast( - cast(left(dsc_subfuncao, 3) as int64) as string - ) as subfuncao, - safe_cast( - cast(left(dsc_programa, 4) as int64) as string - ) as programa, - safe_cast(cast(left(dsc_acao, 4) as int64) as string) as acao, - safe_cast( - replace(left(elemento_despesa, 12), '.', '') as string - ) as elemento_despesa, - round( - safe_cast(valor_empenho_original as float64), 2 - ) as valor_inicial, - round( - safe_cast( - ifnull(safe_cast(valor_reforco as float64), 0) as float64 - ), - 2 - ) as valor_reforco, - round( - safe_cast( - ifnull(safe_cast(valor_anulacao as float64), 0) as float64 - ), - 2 - ) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round( - safe_cast(valor_empenho_original as float64) + safe_cast( - ifnull(safe_cast(valor_reforco as float64), 0) as float64 - ) - - safe_cast( - ifnull(safe_cast(valor_anulacao as float64), 0) as float64 - ), - 2 - ) as valor_final - from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_mg") }} - ), - dlic as ( - select - id_empenho_bd, - case - when (count(distinct id_licitacao)) > 1 then 1 else 0 - end as dlic - from empenhado_mg - group by 1 - ), - empenho_mg as ( - select distinct - e.ano, - e.mes, - e.data, - e.sigla_uf, - e.id_municipio, - e.orgao, - e.id_unidade_gestora, - e.id_licitacao_bd, - case - when dlic = 1 - then (safe_cast(null as string)) - else e.id_licitacao - end as id_licitacao, - e.modalidade_licitacao, - e.id_empenho_bd, - e.id_empenho, - e.numero, - e.descricao, - e.modalidade, - e.funcao, - e.subfuncao, - e.programa, - e.acao, - e.elemento_despesa, - e.valor_inicial, - e.valor_reforco, - e.valor_anulacao, - e.valor_ajuste, - e.valor_final - from empenhado_mg e - left join dlic l on l.id_empenho_bd = e.id_empenho_bd - ), - empenhado_pb as ( - select - safe_cast(dt_ano as int64) as ano, - safe_cast(substring(trim(dt_empenho), -7, 2) as int64) as mes, - safe_cast( - concat( - substring(trim(dt_empenho), -4), - '-', - substring(trim(dt_empenho), -7, 2), - '-', - substring(trim(dt_empenho), 1, 2) - ) as date - ) as data, - 'PB' as sigla_uf, - safe_cast(m.id_municipio as string) as id_municipio, - safe_cast(e.cd_ugestora as string) as orgao, - safe_cast(null as string) as id_unidade_gestora, - safe_cast(null as string) as id_licitacao_bd, - safe_cast(null as string) as id_licitacao, - safe_cast(null as string) as modalidade_licitacao, - safe_cast( - concat( - nu_empenho, - ' ', - e.cd_ugestora, - ' ', - m.id_municipio, - ' ', - (right(dt_ano, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(nu_empenho as string) as numero, - safe_cast(lower(de_historico) as string) as descricao, - safe_cast(null as string) as modalidade, - safe_cast(safe_cast(funcao as int64) as string) as funcao, - safe_cast(safe_cast(subfuncao as int64) as string) as subfuncao, - safe_cast(de_programa as string) as programa, -- substituir por código - safe_cast(de_acao as string) as acao, -- substituir por código - concat( - case - when de_cateconomica = 'Despesa Corrente' - then '3' - when de_cateconomica = 'Despesa de Capital' - then '4' - when de_cateconomica = 'Reserva de Contingência' - then '9' - end, - case - when de_natdespesa = 'Pessoal e Encargos Sociais' - then '1' - when de_natdespesa = 'Juros e Encargos da Dívida' - then '2' - when de_natdespesa = 'Outras Despesas Correntes' - then '3' - when de_natdespesa = 'Investimentos' - then '4' - when de_natdespesa = 'Inversões Financeiras' - then '5' - when de_natdespesa = 'Amortização da Dívida' - then '6' - when de_natdespesa = 'Reserva de Contingência' - then '9' - end, - case - when de_modalidade = 'Transferências à União' - then '20' - when - de_modalidade - = 'Transferências a Instituições Privadas com Fins Lucrativos' - then '30' - when - de_modalidade - = 'Execução Orçamentária Delegada a Estados e ao Distrito Federal' - then '32' - when - de_modalidade - = 'Aplicação Direta §§ 1º e 2º do Art. 24 LC 1412' - then '35' - when de_modalidade = 'Aplicação Direta Art. 25 LC 141' - then '36' - when de_modalidade = 'Transferências a Municípios' - then '40' - when - de_modalidade - = 'Transferências a Municípios – Fundo a Fundo' - then '41' - when - de_modalidade - = 'Transferências a Instituições Privadas sem Fins Lucrativos' - then '50' - when - de_modalidade - = 'Transferências a Instituições Privadas com Fins Lucrativos' - then '60' - when - de_modalidade - = 'Transferências a Instituições Multigovernamentais' - then '70' - when - de_modalidade - = 'Transf. a Consórc Púb. C.Rateio §§ 1º e 2º Art. 24 LC141' - then '71' - when - de_modalidade - = 'Execução Orçamentária Delegada a Consórcios Públicos' - then '72' - when de_modalidade = 'Transferências a Consórcios Públicos' - then '73' - when - de_modalidade - = 'Transf. a Consórc Púb. C.Rateio Art. 25 LC 141' - then '74' - when de_modalidade = 'Transferências ao Exterior' - then '80' - when de_modalidade = 'Aplicações Diretas' - then '90' - when - de_modalidade - = 'Ap. Direta Decor. de Op. entre Órg., Fundos e Ent. Integ. dos Orçamentos Fiscal e da Seguridade Social' - then '91' - when - de_modalidade - = ' Aplicação Direta Decor. de Oper. de Órgãos, Fundos e Entid. Integr. dos Orç. Fiscal e da Seguri. Social com Cons. Públ. do qual o Ente Participe' - then '93' - when - de_modalidade - = ' Aplicação Direta Decor. de Oper. de Órgãos, Fundos e Entid. Integr. dos Orç. Fiscal e da Seguri. Social com Cons. Públ. do qual o Ente Não Participe' - then '94' - else null - end, - cd_elemento - ) as elemento_despesa, - safe_cast(vl_empenho as float64) as valor_inicial - from - {{ set_datalake_project("world_wb_mides_staging.raw_empenho_pb") }} e - left join - {{ - set_datalake_project( - "world_wb_mides_staging.aux_municipio_pb" - ) - }} m on e.cd_ugestora = safe_cast(m.id_unidade_gestora as string) - left join - {{ set_datalake_project("world_wb_mides_staging.aux_funcao") }} f - on e.de_funcao = f.nome_funcao - left join - {{ set_datalake_project("world_wb_mides_staging.aux_subfuncao") }} sf - on e.de_subfuncao = sf.nome_subfuncao - ), - anulacao_pb as ( - select - safe_cast( - concat( - nu_empenho, - ' ', - a.cd_ugestora, - ' ', - m.id_municipio, - ' ', - (right(dt_ano, 2)) - ) as string - ) as id_empenho_bd, - sum(safe_cast(vl_estorno as float64)) as valor_anulacao - from - {{ set_datalake_project("world_wb_mides_staging.raw_estorno_pb") }} a - left join - {{ - set_datalake_project( - "world_wb_mides_staging.aux_municipio_pb" - ) - }} m on a.cd_ugestora = safe_cast(m.id_unidade_gestora as string) - group by 1 - ), - frequencia_pb as ( - select id_empenho_bd, count(id_empenho_bd) as frequencia_id - from empenhado_pb - group by 1 - ), - empenho_completo as ( - select - e.ano, - e.mes, - e.data, - e.sigla_uf, - e.id_municipio, - e.orgao, - e.id_unidade_gestora, - e.id_licitacao_bd, - e.id_licitacao, - e.modalidade_licitacao, - e.id_empenho_bd, - e.id_empenho, - e.numero, - e.descricao, - e.modalidade, - e.funcao, - e.subfuncao, - e.programa, - e.acao, - e.elemento_despesa, - frequencia_id, - round(sum(e.valor_inicial), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_reforco, - round(sum(a.valor_anulacao / frequencia_id), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - from empenhado_pb e - full outer join anulacao_pb a on a.id_empenho_bd = e.id_empenho_bd - left join frequencia_pb f on f.id_empenho_bd = e.id_empenho_bd - group by - 1, - 2, - 3, - 4, - 5, - 6, - 7, - 8, - 9, - 10, - 11, - 12, - 13, - 14, - 15, - 16, - 17, - 18, - 19, - 20, - 21 - ), - empenho_pb as ( - select - e.ano, - e.mes, - e.data, - e.sigla_uf, - e.id_municipio, - e.orgao, - e.id_unidade_gestora, - e.id_licitacao_bd, - e.id_licitacao, - e.modalidade_licitacao, - case - when (frequencia_id > 1) - then (safe_cast(null as string)) - else e.id_empenho_bd - end as id_empenho_bd, - e.id_empenho, - e.numero, - e.descricao, - e.modalidade, - e.funcao, - e.subfuncao, - e.programa, - e.acao, - e.elemento_despesa, - e.valor_inicial, - e.valor_reforco, - e.valor_anulacao, - e.valor_ajuste, - round(e.valor_inicial - ifnull(valor_anulacao, 0), 2) as valor_final - from empenho_completo e - ), - empenho_pe as ( - select - safe_cast(e.anoreferencia as int64) as ano, - (safe_cast(extract(month from date(dataempenho)) as int64)) as mes, - safe_cast( - extract(date from timestamp(dataempenho)) as date - ) as data, - 'PE' as sigla_uf, - safe_cast(codigoibge as string) as id_municipio, - safe_cast(null as string) orgao, - safe_cast(id_unidadegestora as string) as id_unidade_gestora, - safe_cast(null as string) id_licitacao_bd, - safe_cast(null as string) id_licitacao, - safe_cast(null as string) modalidade_licitacao, - safe_cast(null as string) as id_empenho_bd, - safe_cast(trim(id_empenho) as string) as id_empenho, - safe_cast(e.numeroempenho as string) as numero, - safe_cast(lower(historico) as string) as descricao, - safe_cast(left(tipo_empenho, 1) as string) as modalidade, - safe_cast(safe_cast(fun.funcao as int64) as string) as funcao, - safe_cast(safe_cast(sub.subfuncao as int64) as string) as subfuncao, - safe_cast(programa as string) as programa, - safe_cast(codigo_tipo_acao as string) as acao, - concat( - case - when categoria = 'Despesa Corrente' - then '3' - when categoria = 'Despesa de Capital' - then '4' - end, - case - when natureza = 'Pessoal e Encargos Sociais' - then '1' - when natureza = 'Juros e Encargos da Dívida' - then '2' - when natureza = 'Outras Despesas Correntes' - then '3' - when natureza = 'Investimentos' - then '4' - when natureza = 'Inversões Financeiras' - then '5' - when natureza = 'Amortização da Dívida' - then '6' - when natureza = 'Reserva de Contingência' - then '9' - end, - case - when modalidade = 'Transferências à União' - then '20' - when - modalidade - = 'Transferências a Instituições Privadas com Fins Lucrativos' - then '30' - when - modalidade - = 'Execução Orçamentária Delegada a Estados e ao Distrito Federal' - then '32' - when - modalidade - = 'Aplicação Direta à conta de recursos de que tratam os §§ 1o e 2o do art. 24 da Lei Complementar no 141, de 2012' - then '35' - when - modalidade - = 'Aplicação Direta à conta de recursos de que trata o art. 25 da Lei Complementar no 141, de 2012' - then '36' - when modalidade = 'Transferências a Municípios' - then '40' - when - modalidade - = 'Transferências a Municípios – Fundo a Fundo' - then '41' - when - modalidade - = 'Transferências a Instituições Privadas sem Fins Lucrativos' - then '50' - when - modalidade - = 'Transferências a Instituições Privadas com Fins Lucrativos' - then '60' - when - modalidade - = 'Transferências a Instituições Multigovernamentais' - then '70' - when - modalidade - = 'Transferências a Consórcios Públicos mediante contrato de rateio à conta de recursos de que tratam os §§ 1o e 2o do art. 24 da Lei Complementar no 141, de 2012' - then '71' - when - modalidade - = 'Execução Orçamentária Delegada a Consórcios Públicos' - then '72' - when modalidade = 'Transferências a Consórcios Públicos' - then '73' - when modalidade = 'Transferências ao Exterior' - then '80' - when modalidade = 'Aplicações Diretas' - then '90' - when - modalidade - = 'Ap. Direta Decor. de Op. entre Órg., Fundos e Ent. Integ. dos Orçamentos Fiscal e da Seguridade Social' - then '91' - when - modalidade - = ' Aplicação Direta Decor. de Oper. de Órgãos, Fundos e Entid. Integr. dos Orç. Fiscal e da Seguri. Social com Cons. Públ. do qual o Ente Participe' - then '93' - when - modalidade - = ' Aplicação Direta Decor. de Oper. de Órgãos, Fundos e Entid. Integr. dos Orç. Fiscal e da Seguri. Social com Cons. Públ. do qual o Ente Não Participe' - then '94' - else null - end, - case - when elementodespesa = 'Pensões do RPPS e do militar' - then '03' - when elementodespesa = 'Contratação por Tempo Determinado' - then '04' - when - elementodespesa - = 'Outros Benefícios Previdenciários do RPPS' - then '05' - when - elementodespesa - = 'Outros Benefícios Previdenciários do servidor ou do militar' - then '05' - when - elementodespesa - = 'Beneficio Mensal ao Deficiente e ao Idoso' - then '06' - when - elementodespesa - = 'Contribuição a Entidades Fechadas de Previdência' - then '07' - when elementodespesa = 'Outros Benefícios Assistenciais' - then '08' - when - elementodespesa - = 'Outros Benefícios Assistenciais do servidor e do militar' - then '08' - when elementodespesa = 'Salário Família' - then '09' - when elementodespesa = 'Seguro Desemprego e Abono Salarial' - then '10' - when - elementodespesa - = 'Vencimentos e Vantagens Fixas - Pessoal Civil' - then '11' - when - elementodespesa - = 'Vencimentos e Vantagens Fixas - Pessoal Militar' - then '12' - when elementodespesa = 'Obrigações Patronais' - then '13' - when - elementodespesa - = 'Aporte para Cobertura do Déficit Atuarial do RPPS' - then '13' - when elementodespesa = 'Diárias - Civil' - then '14' - when - elementodespesa - = 'Outras Despesas Variáveis - Pessoal Civil' - then '16' - when elementodespesa = 'Auxílio Financeiro a Estudantes' - then '18' - when elementodespesa = 'Auxílio Fardamento' - then '19' - when elementodespesa = 'Auxílio Financeiro a Pesquisadores' - then '20' - when - elementodespesa - = 'Outros Encargos sobre a Dívida por Contrato' - then '22' - when - elementodespesa - = 'Juros, Deságios e Descontos da Dívida Mobiliária' - then '23' - when - elementodespesa - = 'Outros Encargos sobre a Dívida Mobiliária' - then '24' - when - elementodespesa - = 'Encargos sobre Operações de Crédito por Antecipação da Receita' - then '25' - when - elementodespesa - = 'Encargos pela Honra de Avais, Garantias, Seguros e Similares' - then '27' - when - elementodespesa - = 'Remuneração de Cotas de Fundos Autárquicos' - then '28' - when elementodespesa = 'Material de Consumo' - then '30' - when - elementodespesa - = 'Premiações Culturais, Artísticas, Científicas, Desportivas e Outras' - then '31' - when - elementodespesa - = 'Material, Bem ou Serviço para Distribuição Gratuita' - then '32' - when elementodespesa = 'Passagens e Despesas de Locomoção' - then '33' - when - elementodespesa - = 'Outras Despesas de Pessoal decorrentes de Contratos de Terceirização' - then '34' - when elementodespesa = 'Serviços de Consultoria' - then '35' - when elementodespesa = 'Locação de Mão-de-Obra' - then '37' - when - elementodespesa - = 'Outros Serviços de Terceiros ? Pessoa Jurídica' - then '39' - when - elementodespesa - = 'Serviços de Tecnologia da Informação e Comunicação - Pessoa Jurídica' - then '40' - when - elementodespesa - = 'Serviços de Tecnologia da Informação e Comunicação ? Pessoa Jurídica' - then '40' - when elementodespesa = 'Contribuições' - then '41' - when elementodespesa = 'Auxílios' - then '42' - when - elementodespesa - = 'Obrigações Tributárias e Contributivas' - then '47' - when elementodespesa = 'Auxílio-Transporte' - then '49' - when elementodespesa = 'Obras e Instalações' - then '51' - when elementodespesa = 'Equipamentos e Material Permanente' - then '52' - when - elementodespesa = 'Aposentadorias do RGPS ? Área Urbana' - then '54' - when elementodespesa = 'Pensões, exclusiva do RGPS' - then '56' - when - elementodespesa - = 'Outros Benefícios do RGPS ? Área Urbana' - then '58' - when elementodespesa = 'Pensões Especiais' - then '59' - when elementodespesa = 'Aquisição de Imóveis' - then '61' - when - elementodespesa - = 'Constituição ou Aumento de Capital de Empresas' - then '65' - when - elementodespesa - = 'Concessão de Empréstimos e Financiamentos' - then '66' - when elementodespesa = 'Depósitos Compulsórios' - then '67' - when - elementodespesa - = 'Rateio pela Participação em Consórcio Público' - then '70' - when - elementodespesa - = 'Principal da Dívida Contratual Resgatado' - then '71' - when - elementodespesa - = 'Principal da Dívida Mobiliária Resgatado' - then '72' - when - elementodespesa - = 'Correção Monetária ou Cambial da Dívida Contratual Resgatada' - then '73' - when - elementodespesa - = 'Principal Corrigido da Dívida Contratual Refinanciado' - then '77' - when - elementodespesa - = 'Distribuição Constitucional ou Legal de Receitas' - then '81' - when elementodespesa = 'Sentenças Judiciais' - then '91' - when elementodespesa = 'Despesas de Exercícios Anteriores' - then '92' - when elementodespesa = 'Indenizações e Restituições' - then '93' - when - elementodespesa - = 'Indenização pela Execução de Trabalhos de Campo' - then '95' - when - elementodespesa - = 'Ressarcimento de Despesas de Pessoal Requisitado' - then '96' - else null - end - ) as elemento_despesa, - round(safe_cast(0 as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_reforco, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(valorempenhado as float64), 2) as valor_final - from - {{ set_datalake_project("world_wb_mides_staging.raw_empenho_pe") }} e - left join - {{ - set_datalake_project( - "world_wb_mides_staging.aux_municipio_pe" - ) - }} m on e.nomeunidadegestora = m.nomeunidadegestora - left join - {{ set_datalake_project("world_wb_mides_staging.aux_funcao") }} fun - on upper( - trim( - replace( - replace( - e.funcao, 'Encargos Especias', 'Encargos Especiais' - ), - 'Assistêncial Social', - 'Assistência Social' - ) - ) - ) - = upper(nome_funcao) - left join - {{ set_datalake_project("world_wb_mides_staging.aux_subfuncao") }} sub - on upper(trim(e.subfuncao)) = upper(nome_subfuncao) - ), - empenho_pr as ( - select - safe_cast(nranoempenho as int64) as ano, - (safe_cast(extract(month from date(dtempenho)) as int64)) as mes, - safe_cast(extract(date from timestamp(dtempenho)) as date) as data, - 'PR' as sigla_uf, - safe_cast(m.id_municipio as string) as id_municipio, - safe_cast(trim(cdorgao, '0') as string) as orgao, - safe_cast(cdunidade as string) as id_unidade_gestora, - safe_cast(null as string) as id_licitacao_bd, - safe_cast(null as string) as id_licitacao, - safe_cast(null as string) as modalidade_licitacao, - safe_cast( - concat(idempenho, ' ', m.id_municipio) as string - ) as id_empenho_bd, - safe_cast(idempenho as string) as id_empenho, - safe_cast(nrempenho as string) as numero, - safe_cast(lower(dshistorico) as string) as descricao, - safe_cast(left(dstipoempenho, 1) as string) as modalidade, - safe_cast(safe_cast(cdfuncao as int64) as string) as funcao, - safe_cast(safe_cast(cdsubfuncao as int64) as string) as subfuncao, - safe_cast(safe_cast(cdprograma as int64) as string) as programa, - safe_cast(safe_cast(cdprojetoatividade as int64) as string) as acao, - safe_cast( - concat( - cdcategoriaeconomica, - cdgruponatureza, - cdmodalidade, - cdelemento - ) as string - ) as elemento_despesa, - round(safe_cast(vlempenho as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_reforco, - round(safe_cast(vlestornoempenho as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round( - safe_cast(vlempenho as float64) - - ifnull(safe_cast(vlestornoempenho as float64), 0), - 2 - ) as valor_final - from - {{ set_datalake_project("world_wb_mides_staging.raw_empenho_pr") }} e - left join - basedosdados.br_bd_diretorios_brasil.municipio m - on e.cdibge = m.id_municipio_6 - ), - empenhado_rs as ( - select - min(ano_recebimento) as ano_recebimento, - safe_cast(ano_empenho as int64) as ano, - safe_cast(extract(month from date(dt_operacao)) as int64) as mes, - safe_cast( - concat( - substring(dt_operacao, 1, 4), - '-', - substring(dt_operacao, 6, 2), - '-', - substring(dt_operacao, 9, 2) - ) as date - ) as data, - 'RS' as sigla_uf, - safe_cast(a.id_municipio as string) as id_municipio, - safe_cast(c.cd_orgao as string) as orgao, - safe_cast(cd_orgao_orcamentario as string) as id_unidade_gestora, - safe_cast(null as string) as id_licitacao_bd, - safe_cast(null as string) as id_licitacao, - safe_cast(null as string) as modalidade_licitacao, - safe_cast( - concat( - nr_empenho, - ' ', - c.cd_orgao, - ' ', - m.id_municipio, - ' ', - (right(ano_empenho, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(nr_empenho as string) as numero, - safe_cast(lower(historico) as string) as descricao, - safe_cast(null as string) as modalidade, - safe_cast(safe_cast(cd_funcao as int64) as string) as funcao, - safe_cast(safe_cast(cd_subfuncao as int64) as string) as subfuncao, - safe_cast(safe_cast(cd_programa as int64) as string) as programa, - safe_cast(safe_cast(cd_projeto as int64) as string) as acao, - safe_cast( - replace(cd_elemento, '.', '') as string - ) as elemento_despesa, - safe_cast(vl_empenho as float64) as valor_inicial - from - {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} - as c - left join - {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} - as a - on c.cd_orgao = a.cd_orgao - left join - `basedosdados.br_bd_diretorios_brasil.municipio` m - on m.id_municipio = a.id_municipio - where tipo_operacao = 'E' and (safe_cast(vl_empenho as float64) >= 0) - group by - 2, - 3, - 4, - 5, - 6, - 7, - 8, - 9, - 10, - 11, - 12, - 13, - 14, - 15, - 16, - 17, - 18, - 19, - 20, - 21, - 22 - ), - frequencia_rs as ( - select id_empenho_bd, count(id_empenho_bd) as frequencia_id - from empenhado_rs - group by 1 - ), - anulacao_rs as ( - select - safe_cast( - concat( - nr_empenho, - ' ', - c.cd_orgao, - ' ', - m.id_municipio, - ' ', - (right(ano_empenho, 2)) - ) as string - ) as id_empenho_bd, - -1 * sum(safe_cast(vl_empenho as float64)) as valor_anulacao - from - {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} - as c - left join - {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} - as a - on c.cd_orgao = a.cd_orgao - left join - `basedosdados.br_bd_diretorios_brasil.municipio` m - on m.id_municipio = a.id_municipio - where tipo_operacao = 'E' and (safe_cast(vl_empenho as float64) < 0) - group by 1 - ), - empenho_anulacao as ( - select - e.*, - f.frequencia_id, - a.valor_anulacao / f.frequencia_id as valor_anulacao - from empenhado_rs e - left join anulacao_rs a on e.id_empenho_bd = a.id_empenho_bd - left join frequencia_rs f on e.id_empenho_bd = f.id_empenho_bd - ), - dorgao as ( - select - id_empenho_bd, - case when (count(distinct orgao)) > 1 then 1 else 0 end as dorgao - from empenho_anulacao - group by 1 - ), - dugest as ( - select - id_empenho_bd, - case - when (count(distinct id_unidade_gestora)) > 1 then 1 else 0 - end as dugest - from empenho_anulacao - group by 1 - ), - ddesc as ( - select - id_empenho_bd, - case when (count(distinct descricao)) > 1 then 1 else 0 end as ddesc - from empenho_anulacao - group by 1 - ), - dfun as ( - select - id_empenho_bd, - case when (count(distinct funcao)) > 1 then 1 else 0 end as dfun - from empenho_anulacao - group by 1 - ), - dsubf as ( - select - id_empenho_bd, - case when (count(distinct subfuncao)) > 1 then 1 else 0 end as dsubf - from empenho_anulacao - group by 1 - ), - dprog as ( - select - id_empenho_bd, - case when (count(distinct programa)) > 1 then 1 else 0 end as dprog - from empenho_anulacao - group by 1 - ), - dacao as ( - select - id_empenho_bd, - case when (count(distinct acao)) > 1 then 1 else 0 end as dacao - from empenho_anulacao - group by 1 - ), - delem as ( - select - id_empenho_bd, - case - when (count(distinct elemento_despesa)) > 1 then 1 else 0 - end as delem - from empenho_anulacao - group by 1 - ), - dummies as ( - select - o.id_empenho_bd, - dorgao, - dugest, - ddesc, - dfun, - dsubf, - dprog, - dacao, - delem - from dorgao o - left join dugest g on o.id_empenho_bd = g.id_empenho_bd - left join ddesc d on o.id_empenho_bd = d.id_empenho_bd - left join dfun f on o.id_empenho_bd = f.id_empenho_bd - left join dsubf s on o.id_empenho_bd = s.id_empenho_bd - left join dprog p on o.id_empenho_bd = p.id_empenho_bd - left join dacao a on o.id_empenho_bd = a.id_empenho_bd - left join delem e on o.id_empenho_bd = e.id_empenho_bd - ), - empenho_rs as ( - select - min(e.ano) as ano, - min(e.mes) as mes, - min(e.data) as data, - e.sigla_uf, - e.id_municipio, - e.orgao, - e.id_unidade_gestora, - e.id_licitacao_bd, - e.id_licitacao, - e.modalidade_licitacao, - ( - case - when - ( - dorgao = 1 - or dugest = 1 - or dfun = 1 - or dsubf = 1 - or dprog = 1 - or dacao = 1 - or delem = 1 - ) - then (safe_cast(null as string)) - else e.id_empenho_bd - end - ) as id_empenho_bd, - e.id_empenho, - e.numero, - ( - case - when - ( - ddesc = 1 - and ( - dorgao = 0 - or dugest = 0 - or dfun = 0 - or dsubf = 0 - or dprog = 0 - or dacao = 0 - or delem = 0 - ) - ) - then (safe_cast(null as string)) - when - ( - ddesc = 1 - and ( - dorgao = 1 - or dugest = 1 - or dfun = 1 - or dsubf = 1 - or dprog = 1 - or dacao = 1 - or delem = 1 - ) - ) - then (safe_cast(e.descricao as string)) - else e.descricao - end - ) as descricao, - e.modalidade, - e.funcao, - e.subfuncao, - e.programa, - e.acao, - e.elemento_despesa, - round(sum(e.valor_inicial), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_reforco, - round(sum(e.valor_anulacao), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round( - sum(e.valor_inicial) - ifnull(sum(e.valor_anulacao), 0), 2 - ) as valor_final - from empenho_anulacao e - left join dummies d on e.id_empenho_bd = d.id_empenho_bd - group by 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 - ), - empenhado_sp as ( - select - safe_cast(ano_exercicio as int64) as ano, - safe_cast(mes_referencia as int64) as mes, - safe_cast( - concat( - substring(dt_emissao_despesa, -4), - '-', - substring(dt_emissao_despesa, -7, 2), - '-', - substring(dt_emissao_despesa, 1, 2) - ) as date - ) as data, - 'SP' as sigla_uf, - safe_cast(id_municipio as string) as id_municipio, - safe_cast(codigo_orgao as string) as orgao, - safe_cast(null as string) as id_unidade_gestora, - safe_cast(null as string) as id_licitacao_bd, - safe_cast(null as string) as id_licitacao, - case - when ds_modalidade_lic = 'CONVITE' - then '1' - when ds_modalidade_lic = 'TOMADA DE PREÇOS' - then '2' - when ds_modalidade_lic = 'CONCORRÊNCIA' - then '3' - when ds_modalidade_lic = 'PREGÃO' - then '4' - when ds_modalidade_lic = 'Leilão' - then '7' - when ds_modalidade_lic = 'DISPENSA DE LICITAÇÃO' - then '8' - when ds_modalidade_lic = 'BEC-BOLSA ELETRÔNICA DE COMPRAS' - then '9' - when ds_modalidade_lic = 'INEXIGÍVEL' - then '10' - when ds_modalidade_lic = 'CONCURSO' - then '11' - when ds_modalidade_lic = 'RDC' - then '12' - when ds_modalidade_lic = 'OUTROS/NÃO APLICÁVEL' - then '99' - end as modalidade_licitacao, - safe_cast( - concat( - left(nr_empenho, length(nr_empenho) - 5), - ' ', - codigo_orgao, - ' ', - id_municipio, - ' ', - (right(ano_exercicio, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(nr_empenho as string) as numero, - safe_cast(lower(historico_despesa) as string) as descricao, - safe_cast(null as string) as modalidade, - safe_cast(safe_cast(funcao as int64) as string) as funcao, - safe_cast(safe_cast(subfuncao as int64) as string) as subfuncao, - safe_cast(safe_cast(cd_programa as int64) as string) as programa, - safe_cast(safe_cast(cd_acao as int64) as string) as acao, - safe_cast((left(ds_elemento, 8)) as string) as elemento_despesa, - safe_cast(replace(vl_despesa, ',', '.') as float64) as valor_inicial - from - {{ set_datalake_project("world_wb_mides_staging.raw_despesa_sp") }} e - left join - {{ - set_datalake_project( - "world_wb_mides_staging.aux_municipio_sp" - ) - }} m on m.ds_orgao = e.ds_orgao - left join - {{ set_datalake_project("world_wb_mides_staging.aux_funcao") }} - on ds_funcao_governo = upper(nome_funcao) - left join - {{ set_datalake_project("world_wb_mides_staging.aux_subfuncao") }} - on ds_subfuncao_governo = upper(nome_subfuncao) - where tp_despesa = 'Empenhado' - ), - frequencia_sp as ( - select id_empenho_bd, count(id_empenho_bd) as frequencia_id - from empenhado_sp - group by 1 - order by 2 desc - ), - anulacao as ( - select - safe_cast( - concat( - left(nr_empenho, length(nr_empenho) - 5), - ' ', - codigo_orgao, - ' ', - id_municipio, - ' ', - (right(ano_exercicio, 2)) - ) as string - ) as id_empenho_bd, - sum( - safe_cast(replace(vl_despesa, ',', '.') as float64) - ) as valor_anulacao - from - {{ set_datalake_project("world_wb_mides_staging.raw_despesa_sp") }} a - left join - {{ - set_datalake_project( - "world_wb_mides_staging.aux_municipio_sp" - ) - }} m on m.ds_orgao = a.ds_orgao - where tp_despesa = 'Anulação' - group by 1 - ), - reforco as ( - select - safe_cast( - concat( - left(nr_empenho, length(nr_empenho) - 5), - ' ', - codigo_orgao, - ' ', - id_municipio, - ' ', - (right(ano_exercicio, 2)) - ) as string - ) as id_empenho_bd, - sum( - safe_cast(replace(vl_despesa, ',', '.') as float64) - ) as valor_reforco - from - {{ set_datalake_project("world_wb_mides_staging.raw_despesa_sp") }} r - left join - {{ - set_datalake_project( - "world_wb_mides_staging.aux_municipio_sp" - ) - }} m on m.ds_orgao = r.ds_orgao - where tp_despesa = 'Reforço' - group by 1 - ), - empenho_completo_sp as ( - select - e.*, - r.valor_reforco / frequencia_id as valor_reforco, - a.valor_anulacao / frequencia_id as valor_anulacao, - from empenhado_sp e - left join frequencia_sp f on e.id_empenho_bd = f.id_empenho_bd - left join anulacao a on e.id_empenho_bd = a.id_empenho_bd - left join reforco r on e.id_empenho_bd = r.id_empenho_bd - ), - dorgao_sp as ( - select - id_empenho_bd, - case when (count(distinct orgao)) > 1 then 1 else 0 end as dorgao - from empenho_completo_sp - group by 1 - ), - ddesc_sp as ( - select - id_empenho_bd, - case - when (count(distinct ifnull(descricao, ''))) > 1 then 1 else 0 - end as ddesc - from empenho_completo_sp - group by 1 - ), - dmod_sp as ( - select - id_empenho_bd, - case - when (count(distinct modalidade_licitacao)) > 1 then 1 else 0 - end as dmod - from empenho_completo_sp - group by 1 - ), - dfun_sp as ( - select - id_empenho_bd, - case when (count(distinct funcao)) > 1 then 1 else 0 end as dfun - from empenho_completo_sp - group by 1 - ), - dsubf_sp as ( - select - id_empenho_bd, - case when (count(distinct subfuncao)) > 1 then 1 else 0 end as dsubf - from empenho_completo_sp - group by 1 - ), - dprog_sp as ( - select - id_empenho_bd, - case when (count(distinct programa)) > 1 then 1 else 0 end as dprog - from empenho_completo_sp - group by 1 - ), - dacao_sp as ( - select - id_empenho_bd, - case when (count(distinct acao)) > 1 then 1 else 0 end as dacao - from empenho_completo_sp - group by 1 - ), - delem_sp as ( - select - id_empenho_bd, - case - when (count(distinct elemento_despesa)) > 1 then 1 else 0 - end as delem - from empenho_completo_sp - group by 1 - ), - dummies_sp as ( - select - o.id_empenho_bd, - dorgao, - dmod, - ddesc, - dfun, - dsubf, - dprog, - dacao, - delem - from dorgao_sp o - full outer join dmod_sp m on o.id_empenho_bd = m.id_empenho_bd - full outer join ddesc_sp d on o.id_empenho_bd = d.id_empenho_bd - full outer join dfun_sp f on o.id_empenho_bd = f.id_empenho_bd - full outer join dsubf_sp s on o.id_empenho_bd = s.id_empenho_bd - full outer join dprog_sp p on o.id_empenho_bd = p.id_empenho_bd - full outer join dacao_sp a on o.id_empenho_bd = a.id_empenho_bd - full outer join delem_sp e on o.id_empenho_bd = e.id_empenho_bd - ), - empenho_sp as ( - select - min(ano) as ano, - min(mes) as mes, - min(data) as data, - sigla_uf, - id_municipio, - orgao, - id_unidade_gestora, - id_licitacao_bd, - id_licitacao, - modalidade_licitacao, - ( - case - when - ( - dorgao = 1 - or dmod = 1 - or dfun = 1 - or dsubf = 1 - or dprog = 1 - or dacao = 1 - or delem = 1 - ) - then (safe_cast(null as string)) - else e.id_empenho_bd - end - ) as id_empenho_bd, - id_empenho, - numero, - case - when - ( - ddesc = 1 - and ( - dorgao = 0 - or dmod = 0 - or dfun = 0 - or dsubf = 0 - or dprog = 0 - or dacao = 0 - or delem = 0 - ) - ) - then (safe_cast(null as string)) - when - ( - ddesc = 1 - and ( - dorgao = 1 - or dmod = 1 - or dfun = 1 - or dsubf = 1 - or dprog = 1 - or dacao = 1 - or delem = 1 - ) - ) - then (safe_cast(e.descricao as string)) - else e.descricao - end as descricao, - modalidade, - funcao, - subfuncao, - programa, - acao, - elemento_despesa, - round(sum(valor_inicial), 2) as valor_inicial, - round(sum(valor_reforco), 2) as valor_reforco, - round(sum(valor_anulacao), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round( - ifnull(sum(valor_inicial), 0) - + ifnull(sum(valor_reforco), 0) - - ifnull(sum(valor_anulacao), 0), - 2 - ) as valor_final - from empenho_completo_sp e - left join dummies_sp d on d.id_empenho_bd = e.id_empenho_bd - group by 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 - ), - empenho_municipio_sp as ( - select - (safe_cast(exercicio as int64)) as ano, - (safe_cast(extract(month from date(data_empenho)) as int64)) as mes, - safe_cast(data_empenho as date) as data, - 'SP' as sigla_uf, - '3550308' as id_municipio, - safe_cast(codigo_orgao as string) as orgao, - safe_cast(codigo_unidade as string) as id_unidade_gestora, - safe_cast(null as string) as id_licitacao_bd, - safe_cast(null as string) as id_licitacao, - safe_cast(null as string) as modalidade_licitacao, - safe_cast( - concat( - nr_empenho, - ' ', - trim(codigo_orgao), - ' ', - trim(codigo_unidade), - ' ', - '3550308', - ' ', - (right(exercicio, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(id_empenho as string) as id_empenho, - safe_cast(nr_empenho as string) as numero, - safe_cast(observacoes as string) as descricao, - safe_cast( - left( - replace(tipo_empenho, 'Por Estimativa', 'Estimativo'), 1 - ) as string - ) as modalidade, - safe_cast(codigo_funcao as string) as funcao, - safe_cast(codigo_subfuncao as string) as subfuncao, - safe_cast(codigo_programa_governo as string) as programa, - safe_cast(codigo_projeto_atividade as string) as acao, - safe_cast(codigo_conta_despesa as string) as modalidade_despesa, - round(safe_cast(valor_empenho as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_reforco, - round(safe_cast(cancelado as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round( - safe_cast(valor_empenho as float64) - - safe_cast(cancelado as float64), - 2 - ) as valor_final, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_despesa_sp_municipio" - ) - }} - ), - empenhado_municipio_rj_v1 as ( - select - (safe_cast(exercicio_empenho as int64)) as ano, - (safe_cast(extract(month from date(data_empenho)) as int64)) as mes, - safe_cast(data_empenho as date) as data, - 'RJ' as sigla_uf, - '3304557' as id_municipio, - safe_cast(orgao_programa_trabalho as string) as orgao, - safe_cast( - unidade_programa_trabalho as string - ) as id_unidade_gestora, - safe_cast(null as string) as id_licitacao_bd, - safe_cast(n_mero_licita__o as string) as id_licitacao, - case - when modalidade_licitacao = 'Convite' - then '1' - when modalidade_licitacao = 'Tomada De Preços' - then '2' - when modalidade_licitacao = 'Tomada de Preços' - then '2' - when modalidade_licitacao = 'Concorrência' - then '3' - when modalidade_licitacao = 'Pregão' - then '4' - when modalidade_licitacao = 'Leilão' - then '7' - when modalidade_licitacao = 'Dispensa' - then '8' - when modalidade_licitacao = 'Inexigibilidade' - then '10' - when modalidade_licitacao = 'Concurso' - then '11' - when modalidade_licitacao = 'Seleção Pública' - then '31' - when modalidade_licitacao = 'Não Sujeito' - then '99' - end as modalidade_licitacao, - safe_cast( - concat( - nr_empenho, - ' ', - trim(orgao_programa_trabalho), - ' ', - trim(unidade_programa_trabalho), - ' ', - '3304557', - ' ', - (right(exercicio_empenho, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(nr_empenho as string) as numero, - safe_cast(null as string) as descricao, - safe_cast(left(especie, 1) as string) as modalidade, - safe_cast( - cast(substring(programa_trabalho, 7, 2) as int64) as string - ) as funcao, - safe_cast( - cast(substring(programa_trabalho, 10, 3) as int64) as string - ) as subfuncao, - safe_cast( - substring(programa_trabalho, 14, 4) as string - ) as programa, - safe_cast(substring(programa_trabalho, 19, 4) as string) as acao, - safe_cast( - safe_cast(natureza_despesa as int64) as string - ) as modalidade_despesa, - round(safe_cast(valor_empenhado as float64), 2) as valor_final, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_despesa_rj_municipio" - ) - }} - where (safe_cast(exercicio_empenho as int64)) < 2017 - ), - frequencia_rj_v1 as ( - select id_empenho_bd, count(id_empenho_bd) as frequencia_id - from empenhado_municipio_rj_v1 - group by 1 - order by 2 desc - ), - empenho_municipio_rj_v1 as ( - select - e.ano, - e.mes, - e.data, - e.sigla_uf, - e.id_municipio, - e.orgao, - e.id_unidade_gestora, - e.id_licitacao_bd, - e.id_licitacao, - e.modalidade_licitacao, - ( - case - when frequencia_id > 1 - then (safe_cast(null as string)) - else e.id_empenho_bd - end - ) as id_empenho_bd, - e.id_empenho, - e.numero, - e.descricao, - e.modalidade, - e.funcao, - e.subfuncao, - e.programa, - e.acao, - e.modalidade_despesa, - round(safe_cast(0 as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_reforco, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - e.valor_final as valor_final - from empenhado_municipio_rj_v1 e - left join frequencia_rj_v1 f on e.id_empenho_bd = f.id_empenho_bd - ), - empenhado_municipio_rj_v2 as ( - select - (safe_cast(exercicio as int64)) as ano, - (safe_cast(extract(month from date(data)) as int64)) as mes, - safe_cast(data as date) as data, - 'RJ' as sigla_uf, - '3304557' as id_municipio, - safe_cast(ug as string) as orgao, - safe_cast(uo as string) as id_unidade_gestora, - safe_cast(null as string) as id_licitacao_bd, - safe_cast(null as string) as id_licitacao, - case - when licitacao = 'CONVITE' - then '1' - when licitacao = 'TOMADA DE PREÇOS' - then '2' - when licitacao = 'CONCORRÊNCIA' - then '3' - when licitacao = 'PREGÃO' - then '4' - when licitacao = 'PREÇO REGISTRADO/PREGÃO' - then '4' - when licitacao = 'REGISTRO DE PREÇOS EXTERNO/PREGÃO' - then '4' - when licitacao = 'DISPENSA' - then '8' - when licitacao = 'INEXIGIBILIDADE' - then '10' - when licitacao = 'CONCURSO' - then '11' - when licitacao = 'SELEÇÃO PÚBLICA' - then '31' - when licitacao = 'NÃO SUJEITO' - then '99' - end as modalidade_licitacao, - safe_cast( - concat( - left(empenhoexercicio, length(empenhoexercicio) - 5), - ' ', - trim(uo), - ' ', - trim(ug), - ' ', - '3304557', - ' ', - (right(exercicio, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(empenhoexercicio as string) as numero, - safe_cast(historico as string) as descricao, - safe_cast(null as string) as modalidade, - safe_cast(cast(funcao as int64) as string) as funcao, - safe_cast(subfuncao as string) as subfuncao, - safe_cast(programa as string) as programa, - safe_cast(acao as string) as acao, - safe_cast( - concat( - -- categoria econômica - case - when grupo = 'PESSOAL E ENCARGOS SOCIAIS' - then '3' - when grupo = 'JUROS E ENCARGOS DA DIVIDA' - then '3' - when grupo = 'OUTRAS DESPESAS CORRENTES' - then '3' - when grupo = 'INVESTIMENTOS' - then '4' - when grupo = 'INVERSOES FINANCEIRAS' - then '4' - when grupo = 'AMORTIZACAO DA DIVIDA' - then '4' - end, - -- natureza da despesa - case - when grupo = 'PESSOAL E ENCARGOS SOCIAIS' - then '1' - when grupo = 'JUROS E ENCARGOS DA DIVIDA' - then '2' - when grupo = 'OUTRAS DESPESAS CORRENTES' - then '3' - when grupo = 'INVESTIMENTOS' - then '4' - when grupo = 'INVERSOES FINANCEIRAS' - then '5' - when grupo = 'AMORTIZACAO DA DIVIDA' - then '6' - end, - -- modalidade de aplicação - case - when modalidade = 'TRANSFERENCIAS A UNIAO' - then '20' - when - modalidade - = 'TRANSFERENCIAS A ESTADOS E AO DISTRITO FEDERAL' - then '30' - when - modalidade - = 'TRANSFERENCIAS A INSTITUICOES PRIVADAS SEM FINS LUCRATIVOS' - then '50' - when - modalidade - = 'TRANSFERENCIAS A INSTITUICOES PRIVADAS COM FINS LUCRATIVOS' - then '60' - when - modalidade - = 'EXECUCAO DE CONTRATO DE PARCERIA PUBLICO-PRIVADA' - then '67' - when - modalidade - = 'EXECUCAO DE CONTRATO DE PARCERIA PUBLICO-PRIVADA - PPP' - then '67' - when - modalidade - = 'EXECUCAO DE CONTRATO DE PARCERIA PUBLICO PRIVADA - PPP' - then '67' - when - modalidade - = 'DESP. DECORRENTES DA PART. EM FUNDOS, ORGANISMOS OU ENTIDADES ASSEMELHADAS NAC. E INTERN.' - then '84' - when modalidade = 'APLICACOES DIRETAS' - then '90' - when - modalidade - = 'APLIC. DIRETA DECOR. DE OPER. ENTRE ORG., FUNDOS E ENTID. INTEG. DO ORC. FISC. E SEG. SOC.' - then '91' - when - modalidade - = 'APLIC DIRETAS DECOR DE OPER ENTRE ORG, FUNDOS E ENTID INTEGRANTES DOS ORC FISC E SEG SOC' - then '91' - else null - end, - -- elemento e item da despesa - elemento, - subelemento - ) as string - ) as elemento_despesa, - round(safe_cast(valor as float64), 2) as valor_inicial, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_despesa_ato_rj_municipio" - ) - }} - where tipoato = 'EMPENHO' - ), - anulacao_municipio_rj_v2 as ( - select - safe_cast( - concat( - left(empenhoexercicio, length(empenhoexercicio) - 5), - ' ', - trim(uo), - ' ', - trim(ug), - ' ', - '3304557', - ' ', - (right(exercicio, 2)) - ) as string - ) as id_empenho_bd, - round(sum(safe_cast(valor as float64)), 2) as valor_anulacao, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_despesa_ato_rj_municipio" - ) - }} - where tipoato = 'CANCELAMENTO EMPENHO' - group by 1 - ), - empenho_municipio_rj_v2 as ( - select - e.ano, - e.mes, - e.data, - e.sigla_uf, - e.id_municipio, - e.orgao, - e.id_unidade_gestora, - e.id_licitacao_bd, - e.id_licitacao, - e.modalidade_licitacao, - e.id_empenho_bd, - e.id_empenho, - e.numero, - e.descricao, - e.modalidade, - e.funcao, - e.subfuncao, - e.programa, - e.acao, - e.elemento_despesa, - e.valor_inicial as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_reforco, - safe_cast(ifnull(a.valor_anulacao, 0) as float64) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round( - safe_cast( - (e.valor_inicial - ifnull(a.valor_anulacao, 0)) as float64 - ), - 2 - ) as valor_final - from empenhado_municipio_rj_v2 e - left join - anulacao_municipio_rj_v2 a on e.id_empenho_bd = a.id_empenho_bd - ), - empenhado_rj as ( - select - (safe_cast(ano as int64)) as ano, - (safe_cast(extract(month from date(data)) as int64)) as mes, - safe_cast(data as date) as data, - 'RJ' as sigla_uf, - safe_cast(id_municipio as string) as id_municipio, - safe_cast(id_orgao as string) as orgao, - safe_cast(unidade_administrativa as string) as id_unidade_gestora, - safe_cast(null as string) as id_licitacao_bd, - safe_cast(null as string) as id_licitacao, - safe_cast(null as string) as modalidade_licitacao, - safe_cast( - concat( - numero_empenho, - ' ', - id_orgao, - ' ', - id_municipio, - ' ', - (right(ano, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(numero_empenho as string) as numero, - safe_cast(descricao as string) as descricao, - safe_cast(modalidade as string) as modalidade, - safe_cast(cast(funcao as int64) as string) as funcao, - safe_cast(subfuncao as string) as subfuncao, - safe_cast(programa as string) as programa, - safe_cast(atividade as string) as acao, - safe_cast(elemento_despesa as string) as elemento_despesa, - round(safe_cast(valor as float64), 2) as valor_inicial, - from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_rj") }} - where numero_empenho is not null - ), - anulacao_rj as ( - select - safe_cast( - concat( - numero_empenho, - ' ', - id_orgao, - ' ', - id_municipio, - ' ', - (right(ano, 2)) - ) as string - ) as id_empenho_bd, - round(safe_cast(valor as float64), 2) as valor_anulacao, - from - {{ set_datalake_project("world_wb_mides_staging.raw_anulacao_rj") }} - where despesa_liquidada = 'NÃO' and numero_empenho is not null - ), - empenho_rj as ( - select - e.ano, - e.mes, - e.data, - e.sigla_uf, - e.id_municipio, - e.orgao, - e.id_unidade_gestora, - e.id_licitacao_bd, - e.id_licitacao, - e.modalidade_licitacao, - e.id_empenho_bd, - e.id_empenho, - e.numero, - e.descricao, - e.modalidade, - e.funcao, - e.subfuncao, - e.programa, - e.acao, - e.elemento_despesa, - e.valor_inicial as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_reforco, - safe_cast(ifnull(a.valor_anulacao, 0) as float64) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round( - safe_cast( - (e.valor_inicial - ifnull(a.valor_anulacao, 0)) as float64 - ), - 2 - ) as valor_final - from empenhado_rj e - left join anulacao_rj a on e.id_empenho_bd = a.id_empenho_bd - ), - empenho_df as ( - select - (safe_cast(exercicio as int64)) as ano, - (safe_cast(extract(month from date(lancamento)) as int64)) as mes, - safe_cast(lancamento as date) as data, - 'DF' as sigla_uf, - '5300108' as id_municipio, - safe_cast(codigo_ug as string) as orgao, - safe_cast(codigo_gestao as string) as id_unidade_gestora, - safe_cast(null as string) as id_licitacao_bd, - safe_cast(null as string) as id_licitacao, - case - when codigo_licitacao = '1' - then '11' - when codigo_licitacao = '2' - then '1' - when codigo_licitacao = '3' - then '2' - when codigo_licitacao = '4' - then '3' - when codigo_licitacao = '5' - then '8' - when codigo_licitacao = '6' - then '10' - when codigo_licitacao = '7' - then '99' - when codigo_licitacao = '8' - then '32' - when codigo_licitacao = '9' - then '4' - when codigo_licitacao = '10' - then '32' - when codigo_licitacao = '11' - then '31' - when codigo_licitacao = '12' - then '' - when codigo_licitacao = '13' - then '5' - when codigo_licitacao = '14' - then '6' - when codigo_licitacao = '15' - then '5' - when codigo_licitacao = '16' - then '5' - when codigo_licitacao = '17' - then '6' - when codigo_licitacao = '18' - then '3' - when codigo_licitacao = '19' - then '32' - when codigo_licitacao = '20' - then '31' - when codigo_licitacao = '21' - then '31' - when codigo_licitacao = '22' - then '32' - when codigo_licitacao = '23' - then '12' - when codigo_licitacao = '25' - then '98' - when codigo_licitacao = 'INEXIGÍVEL' - then '10' - end as modalidade_licitacao, - safe_cast( - concat( - right(nota_empenho, length(nota_empenho) - 6), - ' ', - codigo_ug, - ' ', - codigo_gestao, - ' ', - '5300108', - ' ', - (right(exercicio, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(nota_empenho as string) as numero, - safe_cast(descricao as string) as descricao, - safe_cast(left(modalidade_empenho, 1) as string) as modalidade, - safe_cast(cast(codigo_funcao as int64) as string) as funcao, - safe_cast(codigo_subfuncao as string) as subfuncao, - safe_cast(codigo_programa as string) as programa, - safe_cast(codigo_acao as string) as acao, - safe_cast(codigo_natureza as string) as elemento_despesa, - round(safe_cast(0 as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_reforco, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round( - safe_cast(replace (valor_final, ',', '.') as float64), 2 - ) as valor_final - from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_df") }} - ), - empenhado_sc as ( - select - safe_cast(ano_emp as int64) as ano, - safe_cast(substring(trim(data_empenho), -7, 2) as int64) as mes, - safe_cast( - concat( - substring(trim(data_empenho), -4), - '-', - substring(trim(data_empenho), -7, 2), - '-', - substring(trim(data_empenho), 1, 2) - ) as date - ) as data, - 'SC' as sigla_uf, - safe_cast(id_municipio as string) as id_municipio, - safe_cast(codigo_orgao as string) as orgao, - safe_cast(null as string) as id_unidade_gestora, - safe_cast(null as string) as id_licitacao_bd, - safe_cast( - case - when - ( - split(nr_licitacao_contrato_convenio, ' / ')[ - offset(0) - ] - ) - != "Sem Licitação" - and ( - split(nr_licitacao_contrato_convenio, ' / ')[ - offset(0) - ] - ) - != "Sem licitação" - and ( - split(nr_licitacao_contrato_convenio, ' / ')[ - offset(0) - ] - ) - != "Sem Licitacao" - and ( - split(nr_licitacao_contrato_convenio, ' / ')[ - offset(0) - ] - ) - != "SEM LICITACAO" - then - ( - split(nr_licitacao_contrato_convenio, ' / ')[ - offset(0) - ] - ) - else '' - end as string - ) as id_licitacao, - safe_cast(null as string) as modalidade_licitacao, - safe_cast( - concat( - num_empenho, - ' ', - codigo_orgao, - ' ', - id_municipio, - ' ', - (right(cast(ano_emp as string), 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(num_empenho as string) as numero, - safe_cast( - lower(descricao_historico_empenho) as string - ) as descricao, - safe_cast(null as string) as modalidade, - safe_cast(cast(left(funcao, 2) as int64) as string) as funcao, - safe_cast(cast(left(subfuncao, 3) as int64) as string) as subfuncao, - safe_cast(null as string) as programa, - safe_cast(null as string) as acao, - safe_cast(elemento_despesa as string) as elemento_despesa, - round(safe_cast(valor_empenho as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_reforco, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(valor_empenho as float64), 2) as valor_final - from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_sc") }} - ), - frequencia_sc as ( - select id_empenho_bd, count(id_empenho_bd) as frequencia_id - from empenhado_sc - group by 1 - order by 2 desc - ), - empenho_sc as ( - select - e.ano, - e.mes, - e.data, - e.sigla_uf, - e.id_municipio, - e.orgao, - e.id_unidade_gestora, - e.id_licitacao_bd, - e.id_licitacao, - e.modalidade_licitacao, - ( - case - when frequencia_id > 1 - then (safe_cast(null as string)) - else e.id_empenho_bd - end - ) as id_empenho_bd, - e.id_empenho, - e.numero, - e.descricao, - e.modalidade, - e.funcao, - e.subfuncao, - e.programa, - e.acao, - e.elemento_despesa, - round(safe_cast(0 as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_reforco, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - e.valor_final as valor_final - from empenhado_sc e - left join frequencia_sc f on e.id_empenho_bd = f.id_empenho_bd - ) - select * - from empenho_mg + from {{ ref("world_wb_mides__empenho_mg") }} union all - (select * from empenho_sp) - union all - (select * from empenho_municipio_sp) - union all - (select * from empenho_pe) - union all - (select * from empenho_pr) + select * + from {{ ref("world_wb_mides__empenho_sp") }} union all - (select * from empenho_rs) + select * + from {{ ref("world_wb_mides__empenho_pe") }} union all - (select * from empenho_pb) + select * + from {{ ref("world_wb_mides__empenho_pr") }} union all - (select * from empenho_ce) + select * + from {{ ref("world_wb_mides__empenho_rs") }} union all - (select * from empenho_rj) + select * + from {{ ref("world_wb_mides__empenho_pb") }} union all - (select * from empenho_municipio_rj_v1) + select * + from {{ ref("world_wb_mides__empenho_ce") }} union all - (select * from empenho_municipio_rj_v2) + select * + from {{ ref("world_wb_mides__empenho_rj") }} union all - (select * from empenho_df) + select * + from {{ ref("world_wb_mides__empenho_df") }} union all - (select * from empenho_sc) + select * + from {{ ref("world_wb_mides__empenho_sc") }} ) diff --git a/models/world_wb_mides/world_wb_mides__liquidacao.sql b/models/world_wb_mides/world_wb_mides__liquidacao.sql index 284fa88f3f..fc4efa581c 100644 --- a/models/world_wb_mides/world_wb_mides__liquidacao.sql +++ b/models/world_wb_mides/world_wb_mides__liquidacao.sql @@ -35,1106 +35,33 @@ select valor_final from ( - with - liquidacao_ce as ( - select - ( - safe_cast(extract(year from date(data_liquidacao)) as int64) - ) as ano, - ( - safe_cast(extract(month from date(data_liquidacao)) as int64) - ) as mes, - safe_cast( - extract(date from timestamp(data_liquidacao)) as date - ) as data, - 'CE' as sigla_uf, - safe_cast(geoibgeid as string) as id_municipio, - safe_cast(codigo_orgao as string) as orgao, - safe_cast(codigo_unidade as string) as id_unidade_gestora, - safe_cast( - concat( - numero_empenho, - ' ', - trim(codigo_orgao), - ' ', - trim(codigo_unidade), - ' ', - geoibgeid, - ' ', - (substring(data_emissao_empenho, 6, 2)), - ' ', - (substring(data_emissao_empenho, 3, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(numero_empenho as string) as numero_empenho, - safe_cast(null as string) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(null as string) as numero, - safe_cast( - nome_responsavel_liquidacao as string - ) as nome_responsavel, - safe_cast( - cpf_responsavel_liquidacao_ as string - ) as documento_responsavel, - safe_cast(null as bool) as indicador_restos_pagar, - round(safe_cast(valor_liquidado as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(valor_liquidado as float64), 2) as valor_final, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_liquidacao_ce" - ) - }} l - left join - {{ - set_datalake_project( - "world_wb_mides_staging.aux_municipio_ce" - ) - }} m on l.codigo_municipio = m.codigo_municipio - ), - liquidacao_mg as ( - select - safe_cast(ano as int64) as ano, - safe_cast(mes as int64) as mes, - safe_cast(data as date) as data, - 'MG' as sigla_uf, - safe_cast(l.id_municipio as string) as id_municipio, - safe_cast(l.orgao as string) as orgao, - safe_cast(l.id_unidade_gestora as string) as id_unidade_gestora, - safe_cast( - ( - case - when id_empenho != '-1' - then - concat( - id_empenho, - ' ', - l.orgao, - ' ', - l.id_municipio, - ' ', - (right(ano, 2)) - ) - when id_empenho = '-1' - then - concat( - id_empenho_origem, - ' ', - r.orgao, - ' ', - r.id_municipio, - ' ', - (right(num_ano_emp_origem, 2)) - ) - end - ) as string - ) as id_empenho_bd, - safe_cast( - ( - case - when id_empenho = '-1' - then replace (id_empenho, '-1', id_empenho_origem) - end - ) as string - ) as id_empenho, - safe_cast(numero_empenho as string) as numero_empenho, - safe_cast( - concat( - id_liquidacao, - ' ', - l.orgao, - ' ', - l.id_municipio, - ' ', - (right(ano, 2)) - ) as string - ) as id_liquidacao_bd, - safe_cast(id_liquidacao as string) as id_liquidacao, - safe_cast(numero_liquidacao as string) as numero, - safe_cast(nome_responsavel as string) as nome_responsavel, - safe_cast(documento_responsavel as string) as documento_responsavel, - safe_cast( - (case when l.id_rsp != '-1' then 1 else 0 end) as bool - ) as indicador_restos_pagar, - round( - safe_cast(valor_liquidacao_original as float64), 2 - ) as valor_inicial, - round(safe_cast(valor_anulado as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round( - safe_cast(valor_liquidacao_original as float64) - - ifnull(safe_cast(valor_anulado as float64), 0), - 2 - ) as valor_final - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_liquidacao_mg" - ) - }} as l - left join - {{ set_datalake_project("world_wb_mides_staging.raw_rsp_mg") }} as r - on l.id_rsp = r.id_rsp - ), - liquidacao_pb as ( - select - safe_cast(dt_ano as int64) as ano, - (safe_cast(substring(dt_liquidacao, -7, 2) as int64)) as mes, - safe_cast( - concat( - substring(dt_liquidacao, -4), - '-', - substring(dt_liquidacao, -7, 2), - '-', - substring(dt_liquidacao, 1, 2) - ) as date - ) as data, - 'PB' as sigla_uf, - safe_cast(id_municipio as string) as id_municipio, - safe_cast(null as string) as orgao, - safe_cast(l.cd_ugestora as string) as id_unidade_gestora, - safe_cast( - concat( - nu_empenho, - ' ', - l.cd_ugestora, - ' ', - m.id_municipio, - ' ', - (right(dt_ano, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(nu_empenho as string) as numero_empenho, - safe_cast(null as string) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(nu_liquidacao as string) as numero, - safe_cast(null as string) as nome_responsavel, - safe_cast(null as string) as documento_responsavel, - safe_cast(null as bool) as indicador_restos_pagar, - round(safe_cast(vl_liquidacao as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(vl_liquidacao as float64), 2) as valor_final, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_liquidacao_pb" - ) - }} l - left join - {{ - set_datalake_project( - "world_wb_mides_staging.aux_municipio_pb" - ) - }} m on l.cd_ugestora = safe_cast(m.id_unidade_gestora as string) - ), - liquidacao_pr as ( - select - safe_cast(nranoliquidacao as int64) as ano, - (safe_cast(extract(month from date(dtliquidacao)) as int64)) as mes, - safe_cast( - extract(date from timestamp(dtliquidacao)) as date - ) as data, - 'PR' as sigla_uf, - safe_cast(id_municipio as string) as id_municipio, - safe_cast(cdorgao as string) as orgao, - safe_cast(cdunidade as string) as id_unidade_gestora, - safe_cast( - concat(l.idempenho, ' ', m.id_municipio) as string - ) as id_empenho_bd, - safe_cast(l.idempenho as string) as id_empenho, - safe_cast(nrempenho as string) as numero_empenho, - safe_cast( - concat(l.idliquidacao, ' ', m.id_municipio) as string - ) as id_liquidacao_bd, - safe_cast(idliquidacao as string) as id_liquidacao, - safe_cast(nrliquidacao as string) as numero, - safe_cast(nmliquidante as string) as nome_responsavel, - safe_cast(nrdocliquidante as string) as documento_responsavel, - safe_cast(null as bool) as indicador_restos_pagar, - round(safe_cast(vlliquidacaobruto as float64), 2) as valor_inicial, - round( - safe_cast(vlliquidacaoestornado as float64), 2 - ) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(vlliquidacaoliquido as float64), 2) as valor_final, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_liquidacao_pr" - ) - }} l - left join - basedosdados.br_bd_diretorios_brasil.municipio m - on cdibge = id_municipio_6 - left join - {{ set_datalake_project("world_wb_mides_staging.raw_empenho_pr") }} e - on l.idempenho = e.idempenho - ), - liquidacao_pe as ( - select - safe_cast(l.anoreferencia as int64) as ano, - (safe_cast(extract(month from date(data)) as int64)) as mes, - safe_cast(extract(date from timestamp(data)) as date) as data, - 'PE' as sigla_uf, - safe_cast(codigoibge as string) as id_municipio, - safe_cast(null as string) orgao, - safe_cast(id_unidadegestora as string) as id_unidade_gestora, - safe_cast(null as string) as id_empenho_bd, - safe_cast(trim(idempenho) as string) as id_empenho, - safe_cast(l.numeroempenho as string) as numero_empenho, - safe_cast(null as string) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(null as string) as numero, - safe_cast(null as string) as nome_responsavel, - safe_cast(null as string) as documento_responsavel, - safe_cast(null as bool) as indicador_restos_pagar, - round(safe_cast(valor as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(valor as float64), 2) as valor_final, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_liquidacao_pe" - ) - }} l - left join - {{ - set_datalake_project( - "world_wb_mides_staging.aux_municipio_pe" - ) - }} m - on l.id_unidade_gestora = safe_cast(m.id_unidadegestora as string) - ), - liquidado_rs as ( - select - min(ano_recebimento) as ano_recebimento, - safe_cast(ano_operacao as int64) as ano, - safe_cast(extract(month from date(dt_operacao)) as int64) as mes, - safe_cast( - concat( - substring(dt_operacao, 1, 4), - '-', - substring(dt_operacao, 6, 2), - '-', - substring(dt_operacao, 9, 2) - ) as date - ) as data, - 'RS' as sigla_uf, - safe_cast(a.id_municipio as string) as id_municipio, - safe_cast(c.cd_orgao as string) as orgao, - safe_cast(cd_orgao_orcamentario as string) as id_unidade_gestora, - safe_cast( - concat( - nr_empenho, - ' ', - c.cd_orgao, - ' ', - m.id_municipio, - ' ', - (right(ano_empenho, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(nr_empenho as string) as numero_empenho, - safe_cast( - concat( - nr_liquidacao, - ' ', - c.cd_orgao, - ' ', - m.id_municipio, - ' ', - (right(ano_empenho, 2)) - ) as string - ) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(nr_liquidacao as string) as numero, - safe_cast(null as string) as nome_responsavel, - safe_cast(null as string) as documento_responsavel, - safe_cast(null as bool) as indicador_restos_pagar, - safe_cast(vl_liquidacao as float64) as valor_inicial - from - {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} - as c - left join - {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} - as a - on c.cd_orgao = a.cd_orgao - left join - `basedosdados.br_bd_diretorios_brasil.municipio` m - on m.id_municipio = a.id_municipio - where tipo_operacao = 'L' and (safe_cast(vl_liquidacao as float64) >= 0) - group by 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18 - ), - estorno_rs as ( - select - safe_cast( - concat( - nr_empenho, - ' ', - c.cd_orgao, - ' ', - m.id_municipio, - ' ', - (right(ano_empenho, 2)) - ) as string - ) as id_empenho_bd, - -1 * sum(safe_cast(vl_liquidacao as float64)) as valor_anulacao - from - {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} - as c - left join - {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} - as a - on c.cd_orgao = a.cd_orgao - left join - `basedosdados.br_bd_diretorios_brasil.municipio` m - on m.id_municipio = a.id_municipio - where tipo_operacao = 'L' and (safe_cast(vl_liquidacao as float64) < 0) - group by 1 - ), - frequencia_rs as ( - select id_empenho_bd, count(id_empenho_bd) as frequencia_id - from liquidado_rs - group by 1 - ), - liquidacao1_rs as ( - select - ano, - mes, - data, - sigla_uf, - id_municipio, - orgao, - id_unidade_gestora, - l.id_empenho_bd, - id_empenho, - numero_empenho, - id_liquidacao_bd, - id_liquidacao, - numero, - nome_responsavel, - documento_responsavel, - indicador_restos_pagar, - sum(valor_inicial) as valor_inicial, - sum(valor_anulacao / frequencia_id) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - sum( - valor_inicial - ifnull((valor_anulacao / frequencia_id), 0) - ) as valor_final - from liquidado_rs l - left join estorno_rs e on l.id_empenho_bd = e.id_empenho_bd - left join frequencia_rs f on l.id_empenho_bd = f.id_empenho_bd - group by 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16 - ), - data_rs as ( - select - id_liquidacao_bd, - case when (count(distinct data)) > 1 then 1 else 0 end as ddata - from liquidacao1_rs - group by 1 - ), - liquidacao_rs as ( - select - ano, - mes, - data, - sigla_uf, - id_municipio, - orgao, - id_unidade_gestora, - id_empenho_bd, - id_empenho, - numero_empenho, - case - when ddata = 1 - then (safe_cast(null as string)) - else l.id_liquidacao_bd - end as id_liquidacao_bd, - id_liquidacao, - numero, - nome_responsavel, - documento_responsavel, - indicador_restos_pagar, - round(valor_inicial, 2), - round(ifnull(valor_anulacao, 0), 2), - valor_ajuste, - round(valor_final, 2) - from liquidacao1_rs l - left join data_rs d on l.id_liquidacao_bd = d.id_liquidacao_bd - ), - liquidado_sp as ( - select - safe_cast(ano_exercicio as int64) as ano, - safe_cast(mes_referencia as int64) as mes, - safe_cast( - concat( - substring(dt_emissao_despesa, -4), - '-', - substring(dt_emissao_despesa, -7, 2), - '-', - substring(dt_emissao_despesa, 1, 2) - ) as date - ) as data, - 'SP' as sigla_uf, - safe_cast(id_municipio as string) as id_municipio, - safe_cast(codigo_orgao as string) as orgao, - safe_cast(null as string) as id_unidade_gestora, - safe_cast( - concat( - left(nr_empenho, length(nr_empenho) - 5), - ' ', - codigo_orgao, - ' ', - id_municipio, - ' ', - (right(ano_exercicio, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(nr_empenho as string) as numero_empenho, - safe_cast( - concat( - left(nr_empenho, length(nr_empenho) - 5), - ' ', - regexp_replace(identificador_despesa, '[^0-9]', ''), - ' ', - codigo_orgao, - ' ', - id_municipio, - ' ', - (right(ano_exercicio, 2)) - ) as string - ) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(null as string) as numero, - safe_cast(null as string) as nome_responsavel, - safe_cast(null as string) as documento_responsavel, - safe_cast(null as bool) as indicador_restos_pagar, - safe_cast(nr_empenho as string) as numero, - case - when ds_modalidade_lic = 'CONVITE' - then '1' - when ds_modalidade_lic = 'TOMADA DE PREÇOS' - then '2' - when ds_modalidade_lic = 'CONCORRÊNCIA' - then '3' - when ds_modalidade_lic = 'PREGÃO' - then '4' - when ds_modalidade_lic = 'Leilão' - then '7' - when ds_modalidade_lic = 'DISPENSA DE LICITAÇÃO' - then '8' - when ds_modalidade_lic = 'BEC-BOLSA ELETRÔNICA DE COMPRAS' - then '9' - when ds_modalidade_lic = 'INEXIGÍVEL' - then '10' - when ds_modalidade_lic = 'CONCURSO' - then '11' - when ds_modalidade_lic = 'RDC' - then '12' - when ds_modalidade_lic = 'OUTROS/NÃO APLICÁVEL' - then '99' - end as modalidade_licitacao, - safe_cast(lower(historico_despesa) as string) as descricao, - safe_cast(null as string) as modalidade, - safe_cast(funcao as string) as funcao, - safe_cast(subfuncao as string) as subfuncao, - safe_cast(cd_programa as string) as programa, - safe_cast(cd_acao as string) as acao, - safe_cast((left(ds_elemento, 8)) as string) as elemento_despesa, - safe_cast(replace(vl_despesa, ',', '.') as float64) as valor_inicial - from - {{ set_datalake_project("world_wb_mides_staging.raw_despesa_sp") }} e - left join - {{ - set_datalake_project( - "world_wb_mides_staging.aux_municipio_sp" - ) - }} m on m.ds_orgao = e.ds_orgao - left join - {{ set_datalake_project("world_wb_mides_staging.aux_funcao") }} - on ds_funcao_governo = upper(nome_funcao) - left join - {{ set_datalake_project("world_wb_mides_staging.aux_subfuncao") }} - on ds_subfuncao_governo = upper(nome_subfuncao) - where tp_despesa = 'Valor Liquidado' - ), - frequencia as ( - select id_empenho_bd, count(id_empenho_bd) as frequencia_id - from liquidado_sp - group by 1 - order by 2 desc - ), - dorgao as ( - select - id_empenho_bd, - case when (count(distinct orgao)) > 1 then 1 else 0 end as dorgao - from liquidado_sp - group by 1 - ), - ddesc as ( - select - id_empenho_bd, - case - when (count(distinct ifnull(descricao, ''))) > 1 then 1 else 0 - end as ddesc - from liquidado_sp - group by 1 - ), - dmod as ( - select - id_empenho_bd, - case - when (count(distinct modalidade_licitacao)) > 1 then 1 else 0 - end as dmod - from liquidado_sp - group by 1 - ), - dfun as ( - select - id_empenho_bd, - case when (count(distinct funcao)) > 1 then 1 else 0 end as dfun - from liquidado_sp - group by 1 - ), - dsubf as ( - select - id_empenho_bd, - case when (count(distinct subfuncao)) > 1 then 1 else 0 end as dsubf - from liquidado_sp - group by 1 - ), - dprog as ( - select - id_empenho_bd, - case when (count(distinct programa)) > 1 then 1 else 0 end as dprog - from liquidado_sp - group by 1 - ), - dacao as ( - select - id_empenho_bd, - case when (count(distinct acao)) > 1 then 1 else 0 end as dacao - from liquidado_sp - group by 1 - ), - delem as ( - select - id_empenho_bd, - case - when (count(distinct elemento_despesa)) > 1 then 1 else 0 - end as delem - from liquidado_sp - group by 1 - ), - dummies as ( - select - o.id_empenho_bd, - dorgao, - dmod, - ddesc, - dfun, - dsubf, - dprog, - dacao, - delem - from dorgao o - full outer join dmod m on o.id_empenho_bd = m.id_empenho_bd - full outer join ddesc d on o.id_empenho_bd = d.id_empenho_bd - full outer join dfun f on o.id_empenho_bd = f.id_empenho_bd - full outer join dsubf s on o.id_empenho_bd = s.id_empenho_bd - full outer join dprog p on o.id_empenho_bd = p.id_empenho_bd - full outer join dacao a on o.id_empenho_bd = a.id_empenho_bd - full outer join delem e on o.id_empenho_bd = e.id_empenho_bd - ), - liquidacao_sp as ( - select - min(ano) as ano, - min(mes) as mes, - min(data) as data, - sigla_uf, - id_municipio, - orgao, - id_unidade_gestora, - ( - case - when - ( - dorgao = 1 - or dmod = 1 - or dfun = 1 - or dsubf = 1 - or dprog = 1 - or dacao = 1 - or delem = 1 - ) - then (safe_cast(null as string)) - else l.id_empenho_bd - end - ) as id_empenho_bd, - id_empenho, - numero_empenho, - id_liquidacao_bd, - id_liquidacao, - safe_cast(null as string) as numero, - nome_responsavel, - documento_responsavel, - indicador_restos_pagar, - round(sum(valor_inicial), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(sum(valor_inicial), 2) as valor_final - from liquidado_sp l - left join dummies d on d.id_empenho_bd = l.id_empenho_bd - group by 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16 - ), - liquidacao_municipio_sp as ( - select - (safe_cast(exercicio as int64)) as ano, - (safe_cast(extract(month from date(data_empenho)) as int64)) as mes, - safe_cast(data_empenho as date) as data, - 'SP' as sigla_uf, - '3550308' as id_municipio, - safe_cast(codigo_orgao as string) as orgao, - safe_cast(codigo_unidade as string) as id_unidade_gestora, - safe_cast( - concat( - nr_empenho, - ' ', - trim(codigo_orgao), - ' ', - trim(codigo_unidade), - ' ', - '3550308', - ' ', - (right(exercicio, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(id_empenho as string) as id_empenho, - safe_cast(nr_empenho as string) as numero_empenho, - safe_cast(null as string) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(null as string) as numero, - safe_cast(null as string) as nome_responsavel, - safe_cast(null as string) as documento_responsavel, - safe_cast(null as bool) as indicador_restos_pagar, - round(safe_cast(liquidado as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(liquidado as float64), 2) as valor_final - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_despesa_sp_municipio" - ) - }} - ), - liquidado_municipio_rj_v1 as ( - select - safe_cast(exercicio_empenho as int64) as ano, - safe_cast(null as int64) as mes, - safe_cast(null as date) as data, - 'RJ' as sigla_uf, - '3304557' as id_municipio, - safe_cast(orgao_programa_trabalho as string) as orgao, - safe_cast( - unidade_programa_trabalho as string - ) as id_unidade_gestora, - safe_cast( - concat( - nr_empenho, - ' ', - trim(orgao_programa_trabalho), - ' ', - trim(unidade_programa_trabalho), - ' ', - '3304557', - ' ', - (right(exercicio_empenho, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(nr_empenho as string) as numero_empenho, - safe_cast(null as string) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(null as string) as numero, - safe_cast(null as string) as nome_responsavel, - safe_cast(null as string) as documento_responsavel, - safe_cast(null as bool) as indicador_restos_pagar, - round(safe_cast(0 as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(valor_liquidado as float64), 2) as valor_final - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_despesa_rj_municipio" - ) - }} - where (safe_cast(exercicio_empenho as int64)) < 2017 - ), - frequencia_rj_v1 as ( - select id_empenho_bd, count(id_empenho_bd) as frequencia_id - from liquidado_municipio_rj_v1 - group by 1 - order by 2 desc - ), - liquidacao_municipio_rj_v1 as ( - select - l.ano, - l.mes, - l.data, - l.sigla_uf, - l.id_municipio, - l.orgao, - l.id_unidade_gestora, - ( - case - when frequencia_id > 1 - then (safe_cast(null as string)) - else l.id_empenho_bd - end - ) as id_empenho_bd, - l.id_empenho, - l.numero_empenho, - l.id_liquidacao_bd, - l.id_liquidacao, - l.numero, - l.nome_responsavel, - l.documento_responsavel, - l.indicador_restos_pagar, - l.valor_inicial, - l.valor_anulacao, - l.valor_ajuste, - l.valor_final - from liquidado_municipio_rj_v1 l - left join frequencia_rj_v1 f on l.id_empenho_bd = f.id_empenho_bd - ), - liquidado_municipio_rj_v2 as ( - select - (safe_cast(exercicio as int64)) as ano, - (safe_cast(extract(month from date(data)) as int64)) as mes, - safe_cast(data as date) as data, - 'RJ' as sigla_uf, - '3304557' as id_municipio, - safe_cast(ug as string) as orgao, - safe_cast(uo as string) as id_unidade_gestora, - safe_cast( - concat( - left(empenhoexercicio, length(empenhoexercicio) - 5), - ' ', - trim(uo), - ' ', - trim(ug), - ' ', - '3304557', - ' ', - (right(exercicio, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(empenhoexercicio as string) as numero_empenho, - safe_cast( - concat( - liquidacao, - ' ', - left(empenhoexercicio, length(empenhoexercicio) - 5), - ' ', - trim(uo), - ' ', - trim(ug), - ' ', - '3304557', - ' ', - (right(exercicio, 2)) - ) as string - ) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(liquidacao as string) as numero, - safe_cast(null as string) as nome_responsavel, - safe_cast(null as string) as documento_responsavel, - safe_cast(null as bool) as indicador_restos_pagar, - round(safe_cast(valor as float64), 2) as valor_inicial - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_despesa_ato_rj_municipio" - ) - }} - where tipoato = 'LIQUIDACAO' - ), - anulacao_municipio_rj_v2 as ( - select - safe_cast(tipoato as string) as tipoato, - safe_cast( - concat( - left(empenhoexercicio, length(empenhoexercicio) - 5), - ' ', - trim(uo), - ' ', - trim(ug), - ' ', - '3304557', - ' ', - (right(exercicio, 2)) - ) as string - ) as id_empenho_bd, - sum(safe_cast(valor as float64)) as valor_anulacao, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_despesa_ato_rj_municipio" - ) - }} - where - tipoato in ( - 'CANCELAMENTO LIQUIDACAO', - 'Cancelamento de liquidação de RPN', - 'CANCELAMENTO DE RPN' - ) - group by 1, 2 - ), - frequencia_rj_v2 as ( - select id_empenho_bd, count(1) as frequencia - from anulacao_municipio_rj_v2 - group by 1 - ), - liquidacao_municipio_rj_v2 as ( - select - l.ano, - l.mes, - l.data, - l.sigla_uf, - l.id_municipio, - l.orgao, - l.id_unidade_gestora, - l.id_empenho_bd, - l.id_empenho, - l.numero_empenho, - l.id_liquidacao_bd, - l.id_liquidacao, - l.numero, - l.nome_responsavel, - l.documento_responsavel, - case - when tipoato = 'Cancelamento de liquidação de RPN' - then true - when tipoato = 'CANCELAMENTO DE RPN' - then true - else false - end as indicador_restos_pagar, - round(safe_cast(l.valor_inicial as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(l.valor_inicial as float64), 2) as valor_final - from liquidado_municipio_rj_v2 l - left join - anulacao_municipio_rj_v2 a on l.id_empenho_bd = a.id_empenho_bd - left join frequencia_rj_v2 f on l.id_empenho_bd = f.id_empenho_bd - ), - liquidacao_rj as ( - select - (safe_cast(ano as int64)) as ano, - (safe_cast(extract(month from date(data)) as int64)) as mes, - safe_cast(data as date) as data, - 'RJ' as sigla_uf, - safe_cast(id_municipio as string) as id_municipio, - safe_cast(id_orgao as string) as orgao, - safe_cast(unidade_administrativa as string) as id_unidade_gestora, - safe_cast( - concat( - numero_empenho, - ' ', - id_orgao, - ' ', - id_municipio, - ' ', - (right(ano, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(numero_empenho as string) as numero_empenho, - safe_cast(null as string) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(null as string) as numero, - safe_cast(null as string) as nome_responsavel, - safe_cast(null as string) as documento_responsavel, - safe_cast(null as bool) as indicador_restos_pagar, - round(safe_cast(valor as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(valor as float64), 2) as valor_final - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_liquidacao_rj" - ) - }} - where numero_empenho is not null - ), - liquidacao_df as ( - select - (safe_cast(exercicio as int64)) as ano, - (safe_cast(extract(month from date(emissao)) as int64)) as mes, - safe_cast(emissao as date) as data, - 'DF' as sigla_uf, - '5300108' as id_municipio, - safe_cast(codigo_ug as string) as orgao, - safe_cast(codigo_gestao as string) as id_unidade_gestora, - safe_cast( - concat( - right(nota_empenho, length(nota_empenho) - 6), - ' ', - codigo_ug, - ' ', - codigo_gestao, - ' ', - '5300108', - ' ', - (right(exercicio, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(nota_empenho as string) as numero_empenho, - case - when length(nota_lancamento) = 11 - then - safe_cast( - concat( - right(nota_lancamento, length(nota_lancamento) - 6), - ' ', - codigo_ug, - ' ', - codigo_gestao, - ' ', - '5300108', - ' ', - (right(exercicio, 2)) - ) as string - ) - end as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(nota_lancamento as string) as numero, - safe_cast(credor as string) as nome_responsavel, - safe_cast(cnpj_cpf_credor as string) as documento_responsavel, - safe_cast(null as bool) as indicador_restos_pagar, - round(safe_cast(0 as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round( - safe_cast(replace(valor, ',', '.') as float64), 2 - ) as valor_inicial, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_liquidacao_df" - ) - }} - ), - liquidado_sc as ( - select - safe_cast(ano_emp as int64) as ano, - safe_cast(substring(trim(data_empenho), -7, 2) as int64) as mes, - safe_cast(null as date) as data, - 'SC' as sigla_uf, - safe_cast(id_municipio as string) as id_municipio, - safe_cast(codigo_orgao as string) as orgao, - safe_cast(null as string) as id_unidade_gestora, - safe_cast( - concat( - num_empenho, - ' ', - codigo_orgao, - ' ', - id_municipio, - ' ', - (right(cast(ano_emp as string), 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(num_empenho as string) as numero_empenho, - safe_cast(null as string) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(null as string) as numero, - safe_cast(null as string) as nome_responsavel, - safe_cast(null as string) as documento_responsavel, - safe_cast(null as bool) as indicador_restos_pagar, - round(safe_cast(0 as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_reforco, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(valor_liquidacao as float64), 2) as valor_final - from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_sc") }} - ), - frequencia_sc as ( - select id_empenho_bd, count(id_empenho_bd) as frequencia_id - from liquidado_sc - group by 1 - order by 2 desc - ), - liquidacao_sc as ( - select - l.ano, - l.mes, - l.data, - l.sigla_uf, - l.id_municipio, - l.orgao, - l.id_unidade_gestora, - ( - case - when frequencia_id > 1 - then (safe_cast(null as string)) - else l.id_empenho_bd - end - ) as id_empenho_bd, - l.id_empenho, - l.numero_empenho, - l.id_liquidacao_bd, - l.id_liquidacao, - l.numero, - l.nome_responsavel, - l.documento_responsavel, - l.indicador_restos_pagar, - l.valor_inicial, - l.valor_anulacao, - l.valor_ajuste, - l.valor_final - from liquidado_sc l - left join frequencia_sc f on l.id_empenho_bd = f.id_empenho_bd - ) - select * - from liquidacao_mg + from {{ ref("world_wb_mides__liquidacao_mg") }} union all - (select * from liquidacao_sp) - union all - (select * from liquidacao_municipio_sp) - union all - (select * from liquidacao_pe) - union all - (select * from liquidacao_pr) + select * + from {{ ref("world_wb_mides__liquidacao_sp") }} union all - (select * from liquidacao_rs) + select * + from {{ ref("world_wb_mides__liquidacao_pe") }} union all - (select * from liquidacao_pb) + select * + from {{ ref("world_wb_mides__liquidacao_pr") }} union all - (select * from liquidacao_ce) + select * + from {{ ref("world_wb_mides__liquidacao_rs") }} union all - (select * from liquidacao_municipio_rj_v1) + select * + from {{ ref("world_wb_mides__liquidacao_pb") }} union all - (select * from liquidacao_municipio_rj_v2) + select * + from {{ ref("world_wb_mides__liquidacao_ce") }} union all - (select * from liquidacao_rj) + select * + from {{ ref("world_wb_mides__liquidacao_rj") }} union all - (select * from liquidacao_df) + select * + from {{ ref("world_wb_mides__liquidacao_df") }} union all - (select * from liquidacao_sc) + select * + from {{ ref("world_wb_mides__liquidacao_sc") }} ) diff --git a/models/world_wb_mides/world_wb_mides__pagamento.sql b/models/world_wb_mides/world_wb_mides__pagamento.sql index 44bf91a58f..0ec3aed236 100644 --- a/models/world_wb_mides/world_wb_mides__pagamento.sql +++ b/models/world_wb_mides/world_wb_mides__pagamento.sql @@ -40,1643 +40,33 @@ select valor_liquido_recebido from ( - with - empenho_ce as ( - select - safe_cast( - concat( - numero_empenho, - ' ', - trim(codigo_orgao), - ' ', - trim(codigo_unidade), - ' ', - m.geoibgeid, - ' ', - (substring(data_emissao_empenho, 6, 2)), - ' ', - (substring(data_emissao_empenho, 3, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(nome_negociante as string) as nome_credor, - safe_cast( - replace ( - replace (numero_documento_negociante, '.', ''), '-', '' - ) as string - ) as documento_credor, - safe_cast(safe_cast(codigo_fonte_ as int64) as string) as fonte, - from - {{ set_datalake_project("world_wb_mides_staging.raw_empenho_ce") }} e - left join - {{ - set_datalake_project( - "world_wb_mides_staging.aux_municipio_ce" - ) - }} m on e.codigo_municipio = m.codigo_municipio - ), - pago_ce as ( - select - ( - safe_cast(extract(year from date(data_nota_pagamento)) as int64) - ) as ano, - ( - safe_cast( - extract(month from date(data_nota_pagamento)) as int64 - ) - ) as mes, - safe_cast( - extract(date from timestamp(data_nota_pagamento)) as date - ) as data, - 'CE' as sigla_uf, - safe_cast(m.geoibgeid as string) as id_municipio, - safe_cast(p.codigo_orgao as string) orgao, - safe_cast(p.codigo_unidade as string) as id_unidade_gestora, - safe_cast( - concat( - p.numero_empenho, - ' ', - trim(p.codigo_orgao), - ' ', - trim(p.codigo_unidade), - ' ', - m.geoibgeid, - ' ', - (substring(p.data_emissao_empenho, 6, 2)), - ' ', - (substring(p.data_emissao_empenho, 3, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(p.numero_empenho as string) as numero_empenho, - safe_cast(null as string) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(null as string) as numero_liquidacao, - safe_cast( - concat( - p.numero_empenho, - ' ', - safe_cast( - safe_cast(numero_nota_pagamento as int64) as string - ), - ' ', - trim(p.codigo_orgao), - ' ', - trim(p.codigo_unidade), - ' ', - m.geoibgeid, - ' ', - (substring(p.data_emissao_empenho, 6, 2)), - ' ', - (substring(p.data_emissao_empenho, 3, 2)) - ) as string - ) as id_pagamento_bd, - safe_cast(null as string) as id_pagamento, - safe_cast(numero_nota_pagamento as string) as numero, - safe_cast(null as bool) as indicador_restos_pagar, - round( - safe_cast(valor_nota_pagamento as float64), 2 - ) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(valor_nota_pagamento as float64), 2) as valor_final, - round(safe_cast(0 as float64), 2) as valor_liquido_recebido, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_pagamento_ce" - ) - }} p - left join - {{ - set_datalake_project( - "world_wb_mides_staging.aux_municipio_ce" - ) - }} m on p.codigo_municipio = m.codigo_municipio - ), - frequencia_ce as ( - select id_pagamento_bd, count(id_pagamento_bd) as frequencia_id - from pago_ce - group by 1 - ), - pagamento_ce as ( - select - ano, - mes, - data, - sigla_uf, - id_municipio, - orgao, - id_unidade_gestora, - p.id_empenho_bd, - p.id_empenho, - p.numero_empenho, - id_liquidacao_bd, - id_liquidacao, - numero_liquidacao, - ( - case - when (frequencia_id > 1) - then (safe_cast(null as string)) - else p.id_pagamento_bd - end - ) as id_pagamento_bd, - id_pagamento, - numero, - nome_credor, - documento_credor, - indicador_restos_pagar, - fonte, - valor_inicial, - valor_anulacao, - valor_ajuste, - valor_final, - valor_liquido_recebido - from pago_ce p - left join frequencia_ce f on p.id_pagamento_bd = f.id_pagamento_bd - left join empenho_ce e on p.id_empenho_bd = e.id_empenho_bd - ), - pagamento_mg as ( - select distinct - safe_cast(p.ano as int64) as ano, - safe_cast(p.mes as int64) as mes, - safe_cast(p.data as date) as data, - safe_cast(p.sigla_uf as string) as sigla_uf, - safe_cast(p.id_municipio as string) as id_municipio, - safe_cast(p.orgao as string) as orgao, - safe_cast(p.id_unidade_gestora as string) as id_unidade_gestora, - safe_cast( - case - when id_empenho != '-1' - then - concat( - id_empenho, - ' ', - p.orgao, - ' ', - p.id_municipio, - ' ', - (right(ano, 2)) - ) - when id_empenho = '-1' - then - concat( - id_empenho_origem, - ' ', - r.orgao, - ' ', - r.id_municipio, - ' ', - (right(num_ano_emp_origem, 2)) - ) - end as string - ) as id_empenho_bd, - safe_cast( - case - when p.id_empenho = '-1' - then replace (p.id_empenho, '-1', id_empenho_origem) - end as string - ) as id_empenho, - safe_cast(p.numero_empenho as string) as numero_empenho, - safe_cast( - case - when p.id_liquidacao != '-1' - then - concat( - p.id_liquidacao, - ' ', - p.orgao, - ' ', - p.id_municipio, - ' ', - (right(p.ano, 2)) - ) - when p.id_liquidacao = '-1' - then - concat( - ' ', - r.orgao, - ' ', - r.id_municipio, - ' ', - (right(p.ano, 2)) - ) - end as string - ) as id_liquidacao_bd, - safe_cast( - case - when p.id_empenho = '-1' - then replace (p.id_liquidacao, '-1', '') - end as string - ) as id_liquidacao, - safe_cast(p.numero_liquidacao as string) as numero_liquidacao, - safe_cast( - concat( - id_pagamento, - ' ', - p.orgao, - ' ', - p.id_municipio, - ' ', - (right(p.ano, 2)) - ) as string - ) as id_pagamento_bd, - safe_cast(id_pagamento as string) as id_pagamento, - safe_cast(p.numero_pagamento as string) as numero, - safe_cast(nome_credor as string) as nome_credor, - safe_cast( - replace(replace (documento_credor, '.', ''), '-', '') as string - ) as documento_credor, - safe_cast( - case when p.id_rsp != '-1' then 1 else 0 end as bool - ) as indicador_restos_pagar, - safe_cast(left(fonte, 3) as string) as fonte, - round( - safe_cast(valor_pagamento_original as float64), 2 - ) as valor_inicial, - round( - ifnull(safe_cast(vlr_anu_fonte as float64), 0), 2 - ) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round( - safe_cast(valor_pagamento_original as float64) - - ifnull(safe_cast(vlr_anu_fonte as float64), 0), - 2 - ) as valor_final, - round( - safe_cast(valor_pagamento_original as float64) - - ifnull(safe_cast(vlr_anu_fonte as float64), 0) - - ifnull(safe_cast(vlr_ret_fonte as float64), 0), - 2 - ) as valor_liquido_recebido, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_pagamento_mg" - ) - }} as p - left join - {{ set_datalake_project("world_wb_mides_staging.raw_rsp_mg") }} as r - on p.id_rsp = r.id_rsp - ), - pago_pb as ( - select - safe_cast(p.dt_ano as int64) as ano, - safe_cast(substring(trim(dt_pagamento), -7, 2) as int64) as mes, - safe_cast( - concat( - substring(trim(dt_pagamento), -4), - '-', - substring(trim(dt_pagamento), -7, 2), - '-', - substring(trim(dt_pagamento), 1, 2) - ) as date - ) as data, - m.sigla_uf, - safe_cast(m.id_municipio as string) as id_municipio, - safe_cast(null as string) as orgao, - safe_cast(p.cd_ugestora as string) as id_unidade_gestora, - safe_cast( - concat( - e.nu_empenho, - ' ', - e.cd_ugestora, - ' ', - m.id_municipio, - ' ', - (right(e.dt_ano, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(p.nu_empenho as string) as numero_empenho, - safe_cast(null as string) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(null as string) as numero_liquidacao, - safe_cast( - concat( - p.nu_empenho, - ' ', - (safe_cast(nu_parcela as int64)), - ' ', - p.cd_ugestora, - ' ', - id_municipio, - ' ', - (right(p.dt_ano, 2)) - ) as string - ) as id_pagamento_bd, - safe_cast(null as string) as id_pagamento, - safe_cast(nu_parcela as string) as numero, - safe_cast(no_credor as string) as nome_credor, - safe_cast( - replace (replace (cd_credor, '.', ''), '-', '') as string - ) as documento_credor, - safe_cast(null as bool) as indicador_restos_pagar, - safe_cast(tp_fonterecursos as string) as fonte, - round(safe_cast(vl_pagamento as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(vl_retencao as float64), 2) as valor_ajuste, - round(safe_cast(vl_pagamento as float64), 2) as valor_final, - round( - safe_cast(vl_pagamento as float64) - - safe_cast(vl_retencao as float64), - 2 - ) as valor_liquido_recebido, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_pagamento_pb" - ) - }} p - left join - {{ set_datalake_project("world_wb_mides_staging.raw_empenho_pb") }} e - on p.nu_empenho = e.nu_empenho - and p.cd_ugestora = e.cd_ugestora - and p.de_uorcamentaria = e.de_uorcamentaria - and p.dt_ano = e.dt_ano - left join - {{ - set_datalake_project( - "world_wb_mides_staging.aux_municipio_pb" - ) - }} m - on safe_cast(e.cd_ugestora as string) - = safe_cast(m.id_unidade_gestora as string) - ), - frequencia_pb as ( - select id_pagamento_bd, count(id_pagamento_bd) frequencia_id - from pago_pb - group by 1 - ), - pagamento_pb as ( - select - ano, - mes, - data, - sigla_uf, - id_municipio, - orgao, - id_unidade_gestora, - case - when (frequencia_id > 1) - then (safe_cast(null as string)) - else p.id_empenho_bd - end as id_empenho_bd, - id_empenho, - numero_empenho, - id_liquidacao_bd, - id_liquidacao, - numero_liquidacao, - case - when (frequencia_id > 1) - then (safe_cast(null as string)) - else p.id_pagamento_bd - end as id_pagamento_bd, - id_pagamento, - numero, - nome_credor, - documento_credor, - indicador_restos_pagar, - fonte, - valor_inicial, - valor_anulacao, - valor_ajuste, - valor_final, - valor_liquido_recebido - from pago_pb p - left join frequencia_pb f on p.id_pagamento_bd = f.id_pagamento_bd - ), - pagamento_pe as ( - select - safe_cast(p.anoreferencia as int64) as ano, - (safe_cast(extract(month from date(data)) as int64)) as mes, - safe_cast(extract(date from timestamp(data)) as date) as data, - safe_cast(unidadefederativa as string) as sigla_uf, - safe_cast(codigoibge as string) as id_municipio, - safe_cast(null as string) orgao, - safe_cast(id_unidadegestora as string) as id_unidade_gestora, - safe_cast(null as string) as id_empenho_bd, - safe_cast(trim(idempenho) as string) as id_empenho, - safe_cast(p.numeroempenho as string) as numero_empenho, - safe_cast(null as string) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(null as string) as numero_liquidacao, - safe_cast(null as string) as id_pagamento_bd, - safe_cast(null as string) as id_pagamento, - safe_cast(null as string) as numero, - safe_cast(null as string) as nome_credor, - safe_cast(null as string) as documento_credor, - safe_cast(null as bool) as indicador_restos_pagar, - safe_cast(null as string) as fonte, - round(safe_cast(valor as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round( - ( - case - when (safe_cast((valor) as float64) < -1000000000000) - then null - else safe_cast((valor) as float64) - end - ), - 2 - ) as valor_final, - round(safe_cast(0 as float64), 2) as valor_liquido_recebido, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_pagamento_pe" - ) - }} p - inner join - {{ - set_datalake_project( - "world_wb_mides_staging.aux_municipio_pe" - ) - }} m - on safe_cast(p.id_unidade_gestora as string) - = safe_cast(m.id_unidadegestora as string) - ), - pagamento_pr as ( - select - safe_cast(nranopagamento as int64) as ano, - (safe_cast(extract(month from date(dtoperacao)) as int64)) as mes, - safe_cast(extract(date from timestamp(dtoperacao)) as date) as data, - sigla_uf, - id_municipio, - safe_cast(cdorgao as string) as orgao, - safe_cast(cdunidade as string) as id_unidade_gestora, - safe_cast( - concat(p.idempenho, ' ', m.id_municipio) as string - ) as id_empenho_bd, - safe_cast(p.idempenho as string) as id_empenho, - safe_cast(nrempenho as string) as numero_empenho, - safe_cast( - concat(p.idliquidacao, ' ', m.id_municipio) as string - ) as id_liquidacao_bd, - safe_cast(p.idliquidacao as string) as id_liquidacao, - safe_cast(null as string) as numero_liquidacao, - safe_cast( - concat(p.idpagamento, ' ', m.id_municipio) as string - ) as id_pagamento_bd, - safe_cast(idpagamento as string) as id_pagamento, - safe_cast(nrpagamento as string) as numero, - safe_cast(nmcredor as string) as nome_credor, - safe_cast( - regexp_replace(nrdoccredor, '[^0-9]', '') as string - ) as documento_credor, - safe_cast(null as bool) as indicador_restos_pagar, - safe_cast(cdfontereceita as string) as fonte, - round(safe_cast(vloperacao as float64), 2) as valor_inicial, - round(safe_cast(nranoliquidacao as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(p.cdibge as float64), 2) as valor_final, - round(safe_cast(0 as float64), 2) as valor_liquido_recebido, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_pagamento_pr" - ) - }} p - left join - {{ set_datalake_project("world_wb_mides_staging.raw_empenho_pr") }} e - on p.idempenho = e.idempenho - left join - basedosdados.br_bd_diretorios_brasil.municipio m - on e.cdibge = id_municipio_6 - ), - pago_rs as ( - select - min(ano_recebimento) as ano_recebimento, - safe_cast(ano_operacao as int64) as ano, - safe_cast(extract(month from date(dt_operacao)) as int64) as mes, - safe_cast( - concat( - substring(dt_operacao, 1, 4), - '-', - substring(dt_operacao, 6, 2), - '-', - substring(dt_operacao, 9, 2) - ) as date - ) as data, - m.sigla_uf as sigla_uf, - safe_cast(a.id_municipio as string) as id_municipio, - safe_cast(c.cd_orgao as string) as orgao, - safe_cast(cd_orgao_orcamentario as string) as id_unidade_gestora, - safe_cast( - concat( - nr_empenho, - ' ', - c.cd_orgao, - ' ', - m.id_municipio, - ' ', - (right(ano_empenho, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(nr_empenho as string) as numero_empenho, - safe_cast( - concat( - nr_empenho, - ' ', - nr_liquidacao, - ' ', - c.cd_orgao, - ' ', - m.id_municipio, - ' ', - (right(ano_empenho, 2)) - ) as string - ) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(nr_liquidacao as string) as numero_liquidacao, - safe_cast( - concat( - nr_empenho, - ' ', - nr_liquidacao, - ' ', - nr_pagamento, - ' ', - c.cd_orgao, - ' ', - m.id_municipio, - ' ', - (right(ano_empenho, 2)) - ) as string - ) as id_pagamento_bd, - safe_cast(null as string) as id_pagamento, - safe_cast(nr_pagamento as string) as numero, - safe_cast(nm_credor as string) as nome_credor, - safe_cast(cnpj_cpf as string) as documento_credor, - safe_cast(null as bool) as indicador_restos_pagar, - safe_cast(null as string) as fonte, - safe_cast(vl_pagamento as float64) as valor_inicial - from - {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} - as c - left join - {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} - as a - on c.cd_orgao = a.cd_orgao - left join - `basedosdados.br_bd_diretorios_brasil.municipio` m - on m.id_municipio = a.id_municipio - where tipo_operacao = 'P' and (safe_cast(vl_pagamento as float64) >= 0) - group by - 2, - 3, - 4, - 5, - 6, - 7, - 8, - 9, - 10, - 11, - 12, - 13, - 14, - 15, - 16, - 17, - 18, - 19, - 20, - 21, - 22 - ), - estorno_rs as ( - select - safe_cast( - concat( - nr_empenho, - ' ', - c.cd_orgao, - ' ', - m.id_municipio, - ' ', - (right(ano_empenho, 2)) - ) as string - ) as id_empenho_bd, - -1 * sum(safe_cast(vl_pagamento as float64)) as valor_anulacao - from - {{ set_datalake_project("world_wb_mides_staging.raw_despesa_rs") }} - as c - left join - {{ set_datalake_project("world_wb_mides_staging.aux_orgao_rs") }} - as a - on c.cd_orgao = a.cd_orgao - left join - `basedosdados.br_bd_diretorios_brasil.municipio` m - on m.id_municipio = a.id_municipio - where tipo_operacao = 'P' and (safe_cast(vl_pagamento as float64) < 0) - group by 1 - ), - frequencia_rs as ( - select id_empenho_bd, count(id_empenho_bd) as frequencia_id - from pago_rs - group by 1 - ), - pagamento1_rs as ( - select - ano, - mes, - data, - sigla_uf, - id_municipio, - orgao, - id_unidade_gestora, - p.id_empenho_bd, - id_empenho, - numero_empenho, - p.id_liquidacao_bd, - id_liquidacao, - numero_liquidacao, - id_pagamento_bd, - id_pagamento, - numero, - nome_credor, - ifnull(documento_credor, '99999999999') as documento_credor, - indicador_restos_pagar, - fonte, - round(sum(valor_inicial), 2) as valor_inicial, - round(sum(valor_anulacao / frequencia_id), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round( - sum( - valor_inicial - ifnull((valor_anulacao / frequencia_id), 0) - ), - 2 - ) as valor_final, - round(safe_cast(0 as float64), 2) as valor_liquido_recebido - from pago_rs p - left join estorno_rs e on p.id_empenho_bd = e.id_empenho_bd - left join frequencia_rs f on p.id_empenho_bd = f.id_empenho_bd - group by - 1, - 2, - 3, - 4, - 5, - 6, - 7, - 8, - 9, - 10, - 11, - 12, - 13, - 14, - 15, - 16, - 17, - 18, - 19, - 20 - ), - ddata_rs as ( - select - id_pagamento_bd, - case when (count(distinct data)) > 1 then 1 else 0 end as ddata - from pagamento1_rs - group by 1 - ), - dorgao_rs as ( - select - id_pagamento_bd, - case when (count(distinct orgao)) > 1 then 1 else 0 end as dorgao - from pagamento1_rs - group by 1 - ), - dugest_rs as ( - select - id_pagamento_bd, - case - when (count(distinct id_unidade_gestora)) > 1 then 1 else 0 - end as dugest - from pagamento1_rs - group by 1 - ), - credor_rs as ( - select - id_pagamento_bd, - case - when (count(distinct nome_credor)) > 1 then 1 else 0 - end as dcredor - from pagamento1_rs - group by 1 - ), - dcredor_rs as ( - select - id_pagamento_bd, - case - when (count(distinct documento_credor)) > 1 then 1 else 0 - end as ddocumento - from pagamento1_rs - group by 1 - ), - dummies as ( - select d.id_pagamento_bd, ddata, dorgao, dugest, dcredor, ddocumento - from ddata_rs d - left join credor_rs c on d.id_pagamento_bd = c.id_pagamento_bd - left join dcredor_rs dc on d.id_pagamento_bd = dc.id_pagamento_bd - left join dorgao_rs o on d.id_pagamento_bd = o.id_pagamento_bd - left join dugest_rs u on d.id_pagamento_bd = u.id_pagamento_bd - ), - pagamento_rs as ( - select - ano, - mes, - data, - sigla_uf, - id_municipio, - orgao, - id_unidade_gestora, - id_empenho_bd, - id_empenho, - numero_empenho, - id_liquidacao_bd, - id_liquidacao, - numero_liquidacao, - case - when - ddata = 1 - or dorgao = 1 - or dugest = 1 - or dcredor = 1 - or ddocumento = 1 - or (numero_liquidacao = '0' and valor_final = 0) - or (numero = '0' and valor_final = 0) - then (safe_cast(null as string)) - else p.id_pagamento_bd - end as id_pagamento_bd, - id_pagamento, - numero, - nome_credor, - documento_credor, - indicador_restos_pagar, - fonte, - valor_inicial, - valor_anulacao, - valor_ajuste, - valor_final, - valor_liquido_recebido - from pagamento1_rs p - left join dummies d on p.id_pagamento_bd = d.id_pagamento_bd - ), - pago_sp as ( - select - safe_cast(ano_exercicio as int64) as ano, - safe_cast(mes_referencia as int64) as mes, - safe_cast( - concat( - substring(dt_emissao_despesa, -4), - '-', - substring(dt_emissao_despesa, -7, 2), - '-', - substring(dt_emissao_despesa, 1, 2) - ) as date - ) as data, - sigla_uf, - safe_cast(id_municipio as string) as id_municipio, - safe_cast(codigo_orgao as string) as orgao, - safe_cast(null as string) as id_unidade_gestora, - safe_cast( - concat( - left(nr_empenho, length(nr_empenho) - 5), - ' ', - codigo_orgao, - ' ', - id_municipio, - ' ', - (right(ano_exercicio, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(nr_empenho as string) as numero_empenho, - safe_cast( - concat( - left(nr_empenho, length(nr_empenho) - 5), - ' ', - regexp_replace(identificador_despesa, '[^0-9]', ''), - ' ', - codigo_orgao, - ' ', - id_municipio, - ' ', - (right(ano_exercicio, 2)) - ) as string - ) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast( - concat( - left(nr_empenho, length(nr_empenho) - 5), - ' ', - regexp_replace(identificador_despesa, '[^0-9]', ''), - ' ', - codigo_orgao, - ' ', - id_municipio, - ' ', - (right(ano_exercicio, 2)) - ) as string - ) as id_pagamento_bd, - safe_cast(null as string) as numero_liquidacao, - safe_cast(null as string) as id_pagamento, - safe_cast(null as string) as numero, - safe_cast(ds_despesa as string) as nome_credor, - safe_cast( - regexp_replace(identificador_despesa, '[^0-9]', '') as string - ) as documento_credor, - safe_cast(null as bool) as indicador_restos_pagar, - safe_cast(null as string) as fonte, - case - when ds_modalidade_lic = 'CONVITE' - then '1' - when ds_modalidade_lic = 'TOMADA DE PREÇOS' - then '2' - when ds_modalidade_lic = 'CONCORRÊNCIA' - then '3' - when ds_modalidade_lic = 'PREGÃO' - then '4' - when ds_modalidade_lic = 'Leilão' - then '7' - when ds_modalidade_lic = 'DISPENSA DE LICITAÇÃO' - then '8' - when ds_modalidade_lic = 'BEC-BOLSA ELETRÔNICA DE COMPRAS' - then '9' - when ds_modalidade_lic = 'INEXIGÍVEL' - then '10' - when ds_modalidade_lic = 'CONCURSO' - then '11' - when ds_modalidade_lic = 'RDC' - then '12' - when ds_modalidade_lic = 'OUTROS/NÃO APLICÁVEL' - then '99' - end as modalidade_licitacao, - safe_cast(lower(historico_despesa) as string) as descricao, - safe_cast(null as string) as modalidade, - safe_cast(funcao as string) as funcao, - safe_cast(subfuncao as string) as subfuncao, - safe_cast(cd_programa as string) as programa, - safe_cast(cd_acao as string) as acao, - safe_cast((left(ds_elemento, 8)) as string) as elemento_despesa, - safe_cast(replace(vl_despesa, ',', '.') as float64) as valor_inicial - from - {{ set_datalake_project("world_wb_mides_staging.raw_despesa_sp") }} e - left join - {{ - set_datalake_project( - "world_wb_mides_staging.aux_municipio_sp" - ) - }} m on m.ds_orgao = e.ds_orgao - left join - {{ set_datalake_project("world_wb_mides_staging.aux_funcao") }} - on ds_funcao_governo = upper(nome_funcao) - left join - {{ set_datalake_project("world_wb_mides_staging.aux_subfuncao") }} - on ds_subfuncao_governo = upper(nome_subfuncao) - where tp_despesa = 'Valor Pago' - ), - frequencia as ( - select id_empenho_bd, count(id_empenho_bd) as frequencia_id - from pago_sp - group by 1 - order by 2 desc - ), - dorgao as ( - select - id_empenho_bd, - case when (count(distinct orgao)) > 1 then 1 else 0 end as dorgao - from pago_sp - group by 1 - ), - ddesc as ( - select - id_empenho_bd, - case - when (count(distinct ifnull(descricao, ''))) > 1 then 1 else 0 - end as ddesc - from pago_sp - group by 1 - ), - dmod as ( - select - id_empenho_bd, - case - when (count(distinct modalidade_licitacao)) > 1 then 1 else 0 - end as dmod - from pago_sp - group by 1 - ), - dfun as ( - select - id_empenho_bd, - case when (count(distinct funcao)) > 1 then 1 else 0 end as dfun - from pago_sp - group by 1 - ), - dsubf as ( - select - id_empenho_bd, - case when (count(distinct subfuncao)) > 1 then 1 else 0 end as dsubf - from pago_sp - group by 1 - ), - dprog as ( - select - id_empenho_bd, - case when (count(distinct programa)) > 1 then 1 else 0 end as dprog - from pago_sp - group by 1 - ), - dacao as ( - select - id_empenho_bd, - case when (count(distinct acao)) > 1 then 1 else 0 end as dacao - from pago_sp - group by 1 - ), - delem as ( - select - id_empenho_bd, - case - when (count(distinct elemento_despesa)) > 1 then 1 else 0 - end as delem - from pago_sp - group by 1 - ), - dcredor as ( - select - id_empenho_bd, - case - when (count(distinct nome_credor)) > 1 then 1 else 0 - end as dcredor - from pago_sp - group by 1 - ), - ddocumento as ( - select - id_empenho_bd, - case - when (count(distinct documento_credor)) > 1 then 1 else 0 - end as ddocumento - from pago_sp - group by 1 - ), - dummies_sp as ( - select - o.id_empenho_bd, - dorgao, - dmod, - ddesc, - dfun, - dsubf, - dprog, - dacao, - delem, - dcredor, - ddocumento - from dorgao o - full outer join dmod m on o.id_empenho_bd = m.id_empenho_bd - full outer join ddesc d on o.id_empenho_bd = d.id_empenho_bd - full outer join dfun f on o.id_empenho_bd = f.id_empenho_bd - full outer join dsubf s on o.id_empenho_bd = s.id_empenho_bd - full outer join dprog p on o.id_empenho_bd = p.id_empenho_bd - full outer join dacao a on o.id_empenho_bd = a.id_empenho_bd - full outer join delem e on o.id_empenho_bd = e.id_empenho_bd - full outer join dcredor c on o.id_empenho_bd = c.id_empenho_bd - full outer join ddocumento dc on o.id_empenho_bd = dc.id_empenho_bd - ), - frequencia_pg_sp as ( - select id_pagamento_bd, count(id_pagamento_bd) frequencia_id - from pago_sp - group by 1 - ), - pagamento_sp as ( - select - min(ano) as ano, - min(mes) as mes, - min(data) as data, - sigla_uf, - id_municipio, - orgao, - id_unidade_gestora, - ( - case - when - ( - dorgao = 1 - or dmod = 1 - or dfun = 1 - or dsubf = 1 - or dprog = 1 - or dacao = 1 - or delem = 1 - ) - then (safe_cast(null as string)) - else p.id_empenho_bd - end - ) as id_empenho_bd, - id_empenho, - numero_empenho, - ( - case - when - ( - dorgao = 1 - or dmod = 1 - or dfun = 1 - or dsubf = 1 - or dprog = 1 - or dacao = 1 - or delem = 1 - ) - then (safe_cast(null as string)) - else p.id_liquidacao_bd - end - ) as id_liquidacao_bd, - id_liquidacao, - numero_liquidacao, - ( - case - when - ( - dorgao = 1 - or dmod = 1 - or dfun = 1 - or dsubf = 1 - or dprog = 1 - or dacao = 1 - or delem = 1 - or dcredor = 1 - or ddocumento = 1 - ) - or frequencia_id > 1 - then (safe_cast(null as string)) - else p.id_pagamento_bd - end - ) as id_pagamento_bd, - id_pagamento, - numero, - nome_credor, - documento_credor, - indicador_restos_pagar, - fonte, - round(safe_cast(sum(valor_inicial) as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(sum(valor_inicial) as float64), 2) as valor_final, - round(safe_cast(0 as float64), 2) as valor_liquido_recebido - from pago_sp p - left join dummies_sp d on d.id_empenho_bd = p.id_empenho_bd - left join frequencia_pg_sp f on f.id_pagamento_bd = p.id_pagamento_bd - group by 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 - ), - pagamento_municipio_sp as ( - select - (safe_cast(exercicio as int64)) as ano, - (safe_cast(extract(month from date(data_empenho)) as int64)) as mes, - safe_cast(data_empenho as date) as data, - 'SP' as sigla_uf, - '3550308' as id_municipio, - safe_cast(codigo_orgao as string) as orgao, - safe_cast(codigo_unidade as string) as id_unidade_gestora, - safe_cast( - concat( - nr_empenho, - ' ', - trim(codigo_orgao), - ' ', - trim(codigo_unidade), - ' ', - '3550308', - ' ', - (right(exercicio, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(id_empenho as string) as id_empenho, - safe_cast(nr_empenho as string) as numero_empenho, - safe_cast(null as string) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(null as string) as numero_liquidacao, - safe_cast(null as string) as id_pagamento_bd, - safe_cast(null as string) as id_pagamento, - safe_cast(null as string) as numero, - safe_cast(razao_social as string) as nome_credor, - safe_cast(cpf_cnpj as string) as documento_credor, - safe_cast(null as bool) as indicador_restos_pagar, - safe_cast(codigo_fonte_recurso as string) as fonte, - round(safe_cast(pago as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(pago as float64), 2) as valor_final, - round(safe_cast(pago as float64), 2) as valor_liquido_recebido, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_despesa_sp_municipio" - ) - }} - ), - pago_municipio_rj_v1 as ( - select - safe_cast(exercicio_empenho as int64) as ano, - safe_cast(null as int64) as mes, - safe_cast(null as date) as data, - 'RJ' as sigla_uf, - '3304557' as id_municipio, - safe_cast(orgao_programa_trabalho as string) as orgao, - safe_cast( - unidade_programa_trabalho as string - ) as id_unidade_gestora, - safe_cast( - concat( - nr_empenho, - ' ', - trim(orgao_programa_trabalho), - ' ', - trim(unidade_programa_trabalho), - ' ', - '3304557', - ' ', - (right(exercicio_empenho, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(nr_empenho as string) as numero_empenho, - safe_cast(null as string) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(null as string) as numero_liquidacao, - safe_cast(null as string) as id_pagamento_bd, - safe_cast(null as string) as id_pagamento, - safe_cast(null as string) as numero, - safe_cast(favorecido as string) as nome_credor, - safe_cast(codigo_favorecido as string) as documento_credor, - safe_cast(null as bool) as indicador_restos_pagar, - safe_cast(fonte_recursos as string) as fonte, - round(safe_cast(0 as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(valor_pago as float64), 2) as valor_final, - round(safe_cast(valor_pago as float64), 2) as valor_liquido_recebido - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_despesa_rj_municipio" - ) - }} - ), - frequencia_rj_v1 as ( - select id_empenho_bd, count(id_empenho_bd) as frequencia_id - from pago_municipio_rj_v1 - group by 1 - order by 2 desc - ), - pagamento_municipio_rj_v1 as ( - select - p.ano, - p.mes, - p.data, - p.sigla_uf, - p.id_municipio, - p.orgao, - p.id_unidade_gestora, - ( - case - when frequencia_id > 1 - then (safe_cast(null as string)) - else p.id_empenho_bd - end - ) as id_empenho_bd, - p.id_empenho, - p.numero_empenho, - p.id_liquidacao_bd, - p.id_liquidacao, - p.numero_liquidacao, - p.id_pagamento_bd, - p.id_pagamento, - p.numero, - p.nome_credor, - p.documento_credor, - p.indicador_restos_pagar, - p.fonte, - p.valor_inicial, - p.valor_anulacao, - p.valor_ajuste, - p.valor_final, - p.valor_liquido_recebido - from pago_municipio_rj_v1 p - left join frequencia_rj_v1 f on p.id_empenho_bd = f.id_empenho_bd - ), - pago_municipio_rj_v2 as ( - select - (safe_cast(exercicio as int64)) as ano, - (safe_cast(extract(month from date(data)) as int64)) as mes, - safe_cast(data as date) as data, - 'RJ' as sigla_uf, - '3304557' as id_municipio, - safe_cast(ug as string) as orgao, - safe_cast(uo as string) as id_unidade_gestora, - safe_cast( - concat( - left(empenhoexercicio, length(empenhoexercicio) - 5), - ' ', - trim(uo), - ' ', - trim(ug), - ' ', - '3304557', - ' ', - (right(exercicio, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(empenhoexercicio as string) as numero_empenho, - safe_cast( - concat( - liquidacao, - ' ', - left(empenhoexercicio, length(empenhoexercicio) - 5), - ' ', - trim(uo), - ' ', - trim(ug), - ' ', - '3304557', - ' ', - (right(exercicio, 2)) - ) as string - ) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(liquidacao as string) as numero_liquidacao, - safe_cast( - concat( - pagamento, - ' ', - left(empenhoexercicio, length(empenhoexercicio) - 5), - ' ', - trim(uo), - ' ', - trim(ug), - ' ', - '3304557', - ' ', - (right(exercicio, 2)) - ) as string - ) as id_pagamento_bd, - safe_cast(null as string) as id_pagamento, - safe_cast(pagamento as string) as numero, - safe_cast(nomecredor as string) as nome_credor, - safe_cast(credor as string) as documento_credor, - safe_cast(null as bool) as indicador_restos_pagar, - safe_cast(fonterecursos as string) as fonte, - round(safe_cast(valor as float64), 2) as valor_inicial, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_despesa_ato_rj_municipio" - ) - }} - where tipoato = 'PAGAMENTO' - ), - anulacao_municipio_rj_v2 as ( - select - safe_cast(tipoato as string) as tipoato, - safe_cast( - concat( - left(empenhoexercicio, length(empenhoexercicio) - 5), - ' ', - trim(uo), - ' ', - trim(ug), - ' ', - '3304557', - ' ', - (right(exercicio, 2)) - ) as string - ) as id_empenho_bd, - sum(safe_cast(valor as float64)) as valor_anulacao, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_despesa_ato_rj_municipio" - ) - }} - where - tipoato in ( - 'CANCEL.PAGAMENTO RET.DOTAÇÃO', - 'CANCEL.PAGAMENTO RET.EMPENHO', - 'CANCELAMENTO DE PAGAMENTO DE RPP', - 'CANCELAMENTO DE PAGAMENTO DE RPN', - 'Cancelamento de RPP' - ) - group by 1, 2 - ), - frequencia_rj_v2 as ( - select id_empenho_bd, count(1) as frequencia - from anulacao_municipio_rj_v2 - group by 1 - ), - pagamento_municipio_rj_v2 as ( - select - p.ano, - p.mes, - p.data, - p.sigla_uf, - p.id_municipio, - p.orgao, - p.id_unidade_gestora, - p.id_empenho_bd, - p.id_empenho, - p.numero_empenho, - p.id_liquidacao_bd, - p.id_liquidacao, - p.numero_liquidacao, - p.id_empenho_bd, - p.id_empenho, - p.numero, - p.nome_credor, - p.documento_credor, - case - when tipoato = 'CANCELAMENTO DE PAGAMENTO DE RPP' - then true - when tipoato = 'CANCELAMENTO DE PAGAMENTO DE RPN' - then true - when tipoato = 'Cancelamento de RPP' - then true - else false - end as indicador_restos_pagar, - p.fonte, - round(safe_cast(p.valor_inicial as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(p.valor_inicial as float64), 2) as valor_final, - round( - safe_cast(p.valor_inicial as float64), 2 - ) as valor_liquido_recebido - from pago_municipio_rj_v2 p - left join - anulacao_municipio_rj_v2 a on p.id_empenho_bd = a.id_empenho_bd - left join frequencia_rj_v2 f on p.id_empenho_bd = f.id_empenho_bd - ), - pagamento_rj as ( - select - (safe_cast(ano as int64)) as ano, - (safe_cast(extract(month from date(data)) as int64)) as mes, - safe_cast(data as date) as data, - 'RJ' as sigla_uf, - safe_cast(id_municipio as string) as id_municipio, - safe_cast(id_orgao as string) as orgao, - safe_cast(unidade_administrativa as string) as id_unidade_gestora, - safe_cast( - concat( - numero_empenho, - ' ', - id_orgao, - ' ', - id_municipio, - ' ', - (right(ano, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(numero_empenho as string) as numero_empenho, - safe_cast(null as string) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(null as string) as numero_liquidacao, - safe_cast(null as string) as id_pagamento_bd, - safe_cast(null as string) as id_pagamento, - safe_cast(null as string) as numero, - safe_cast(credor as string) as nome_credor, - safe_cast(null as string) as documento_credor, - safe_cast(null as bool) as indicador_restos_pagar, - safe_cast(fonte as string) as fonte, - round(safe_cast(valor as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(valor as float64), 2) as valor_final, - round(safe_cast(valor as float64), 2) as valor_liquido_recebido, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_pagamento_rj" - ) - }} - where numero_empenho is not null - ), - pagamento_df as ( - select - (safe_cast(exercicio as int64)) as ano, - safe_cast(substring(emissao, -7, 2) as int64) as mes, - safe_cast( - concat( - substring(emissao, -4), - '-', - substring(emissao, -7, 2), - '-', - substring(emissao, 1, 2) - ) as date - ) as data, - 'DF' as sigla_uf, - '5300108' as id_municipio, - safe_cast(codigo_ug as string) as orgao, - safe_cast(codigo_gestao as string) as id_unidade_gestora, - safe_cast( - concat( - right(nota_empenho, length(nota_empenho) - 6), - ' ', - codigo_ug, - ' ', - codigo_gestao, - ' ', - '5300108', - ' ', - (right(exercicio, 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(nota_empenho as string) as numero_empenho, - case - when length(nota_lancamento) = 11 - then - safe_cast( - concat( - right(nota_lancamento, length(nota_lancamento) - 6), - ' ', - codigo_ug, - ' ', - codigo_gestao, - ' ', - '5300108', - ' ', - (right(exercicio, 2)) - ) as string - ) - end as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(nota_lancamento as string) as numero_liquidacao, - case - when length(numero_ordem_bancaria) = 11 - then - safe_cast( - concat( - right( - numero_ordem_bancaria, - length(numero_ordem_bancaria) - 6 - ), - ' ', - codigo_ug, - ' ', - codigo_gestao, - ' ', - '5300108', - ' ', - (right(exercicio, 2)) - ) as string - ) - end as id_pagamento_bd, - safe_cast(null as string) as id_pagamento, - safe_cast(numero_ordem_bancaria as string) as numero, - safe_cast(credor as string) as nome_credor, - safe_cast(cnpj_cpf_credor as string) as documento_credor, - case - when ano_ordem_bancaria != ano_nota_empenho then true else false - end as indicador_restos_pagar, - safe_cast(null as string) as fonte, - round( - safe_cast(replace(valor_final_x, ',', '.') as float64), 2 - ) as valor_inicial, - round( - safe_cast(replace(valor_cancelado, ',', '.') as float64), 2 - ) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round( - safe_cast(replace(valor_final_x, ',', '.') as float64) - - safe_cast(replace(valor_cancelado, ',', '.') as float64), - 2 - ) as valor_final, - round( - safe_cast(replace(valor_final_x, ',', '.') as float64) - - safe_cast(replace(valor_cancelado, ',', '.') as float64), - 2 - ) as valor_liquido_recebido, - from - {{ - set_datalake_project( - "world_wb_mides_staging.raw_pagamento_df" - ) - }} - ), - pago_sc as ( - select - safe_cast(ano_emp as int64) as ano, - safe_cast(substring(trim(data_empenho), -7, 2) as int64) as mes, - safe_cast(null as date) as data, - 'SC' as sigla_uf, - safe_cast(id_municipio as string) as id_municipio, - safe_cast(codigo_orgao as string) as orgao, - safe_cast(null as string) as id_unidade_gestora, - safe_cast( - concat( - num_empenho, - ' ', - codigo_orgao, - ' ', - id_municipio, - ' ', - (right(cast(ano_emp as string), 2)) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - safe_cast(num_empenho as string) as numero_empenho, - safe_cast(null as string) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - safe_cast(null as string) as numero_liquidacao, - safe_cast(null as string) as id_pagamento_bd, - safe_cast(null as string) as id_pagamento, - safe_cast(null as string) as numero, - safe_cast(nome_credor as string) as nome_credor, - safe_cast(cpf_cnpj as string) as documento_credor, - safe_cast(null as bool) as indicador_restos_pagar, - safe_cast(right(especificacao_fonte_recurso, 2) as string) as fonte, - round(safe_cast(0 as float64), 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(safe_cast(valor_pagamento as float64), 2) as valor_final, - round( - safe_cast(valor_pagamento as float64), 2 - ) as valor_liquido_recebido - from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_sc") }} - ), - frequencia_sc as ( - select id_empenho_bd, count(id_empenho_bd) as frequencia_id - from pago_sc - group by 1 - order by 2 desc - ), - pagamento_sc as ( - select - p.ano, - p.mes, - p.data, - p.sigla_uf, - p.id_municipio, - p.orgao, - p.id_unidade_gestora, - ( - case - when frequencia_id > 1 - then (safe_cast(null as string)) - else p.id_empenho_bd - end - ) as id_empenho_bd, - p.id_empenho, - p.numero_empenho, - p.id_liquidacao_bd, - p.id_liquidacao, - p.numero_liquidacao, - p.id_pagamento_bd, - p.id_pagamento, - p.numero, - p.nome_credor, - p.documento_credor, - p.indicador_restos_pagar, - p.fonte, - p.valor_inicial, - p.valor_anulacao, - p.valor_ajuste, - p.valor_final, - p.valor_liquido_recebido - from pago_sc p - left join frequencia_sc f on p.id_empenho_bd = f.id_empenho_bd - ) - select * - from pagamento_mg + from {{ ref("world_wb_mides__pagamento_mg") }} union all - (select * from pagamento_sp) - union all - (select * from pagamento_municipio_sp) - union all - (select * from pagamento_pe) - union all - (select * from pagamento_pr) + select * + from {{ ref("world_wb_mides__pagamento_sp") }} union all - (select * from pagamento_rs) + select * + from {{ ref("world_wb_mides__pagamento_pe") }} union all - (select * from pagamento_pb) + select * + from {{ ref("world_wb_mides__pagamento_pr") }} union all - (select * from pagamento_ce) + select * + from {{ ref("world_wb_mides__pagamento_rs") }} union all - (select * from pagamento_municipio_rj_v1) + select * + from {{ ref("world_wb_mides__pagamento_pb") }} union all - (select * from pagamento_municipio_rj_v2) + select * + from {{ ref("world_wb_mides__pagamento_ce") }} union all - (select * from pagamento_rj) + select * + from {{ ref("world_wb_mides__pagamento_rj") }} union all - (select * from pagamento_df) + select * + from {{ ref("world_wb_mides__pagamento_df") }} union all - (select * from pagamento_sc) + select * + from {{ ref("world_wb_mides__pagamento_sc") }} ) From aae97e795324167af10fa1863591429bf2018c45 Mon Sep 17 00:00:00 2001 From: Ricardo Dahis Date: Tue, 25 Aug 2026 15:41:06 +1000 Subject: [PATCH 3/6] fix(ci): only treat models//.sql as a table in table-approve table-approve reads dataset_id off a changed file's parent directory, so any model nested one level deeper yields the subdirectory name as the dataset. For the per-state models this repo now has under models/world_wb_mides/states/, that means launching a materialization flow for a dataset literally called `states` -- 30 of them -- and the loop raises on the first failure, which can abort before the real tables build. Simulated against the MiDES branch: 36 flow runs under the old rule, 6 under the new one. Restricting to depth 3 also fixes a latent case already in the tree: models/br_inpe_queimadas/code/microdados_old/br_inpe_queimadas__microdados_old.sql would parse as dataset_id `microdados_old`. All 1046 real models sit at models//.sql, so nothing that should materialize stops materializing. A subdirectory under a dataset holds intermediate models with no table of their own (the ephemeral per-state models compile into their parent) or archived SQL kept for reference -- neither is a published table. This is the same class of exclusion the function already applies to macros/, tests-dbt/ and analysis/. Committed with SKIP=pyrefly-check: that hook fails identically on untouched files in a git worktree, because the worktree lives under .claude/ and its include pattern therefore matches no Python files. Unrelated to this change. --- .github/workflows/scripts/prefect_run_dbt.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/.github/workflows/scripts/prefect_run_dbt.py b/.github/workflows/scripts/prefect_run_dbt.py index 5bb7e5730d..3eae317845 100644 --- a/.github/workflows/scripts/prefect_run_dbt.py +++ b/.github/workflows/scripts/prefect_run_dbt.py @@ -85,10 +85,21 @@ def get_datasets_tables_from_modified_files( # generic test or macro, not a table. Treating it as a phantom model would # abort prod materialization before any real table builds, so restrict to # models. + # + # A published table is exactly `models//.sql`. Anything + # deeper is not one: `dataset_id` is read off the parent directory, so + # `models///.sql` would yield `` as the + # dataset and launch a materialization flow for a dataset that does not + # exist. Subdirectories hold intermediate models (e.g. the ephemeral + # per-state models under world_wb_mides/states/, which compile into their + # parent and have no table of their own) and archived SQL kept for + # reference (br_inpe_queimadas/code/). Skip both. sql_files: list[Path] = [ file for file in modified_files - if file.suffix == ".sql" and file.parts and file.parts[0] == "models" + if file.suffix == ".sql" + and len(file.parts) == 3 + and file.parts[0] == "models" ] datasets_tables: list[tuple[str, str, bool, bool]] = [ From b3862c0c333bf3b7b364c0186e947a63d634e625 Mon Sep 17 00:00:00 2001 From: Ricardo Dahis Date: Tue, 25 Aug 2026 16:10:41 +1000 Subject: [PATCH 4/6] feat(world_wb_mides): add Tocantins to empenho, liquidacao and pagamento TO's raw data has sat in gs://basedosdados-dev/staging/world_wb_mides/ since May 2024 -- 9.0 GB across three tables, 2013-2022, all 139 TO municipalities -- with no dbt model, so it reached neither dev nor prod despite the MiDES documentation site listing the state as covered. This is the first of the five staged-but-unmodelled states. TCE-TO publishes a movement ledger: one row per movement on an empenho, liquidacao or pagamento, signed by `sinal` ('+' issue, '-' anulacao). The MiDES schema wants one row per line with the movements folded into valor_inicial / valor_anulacao / valor_final, so each model aggregates by sinal to its line key. Emitting one row per movement would have put anulacao amounts into valor_inicial. The line key is (municipio, orgao, numero, ..., rubrica), not the document number alone. Of the 238,192 empenho numbers carrying more than one '+' row, 228,729 (96%) differ in `rubrica` -- separate budget lines of one empenho, not reforcos. Keying on the number alone would merge distinct lines and attribute each line's anulacao to all of them. Three source-driven decisions worth recording: - **modalidade_licitacao is remapped, not passed through.** TCE-TO's codes do not match the MiDES dictionary -- TO 1 = Dispensa where MiDES 1 = Convite -- so passing them through would have silently corrupted the column for every TO row. Mapped by description onto world_wb_mides.dicionario. TO's "Registro de Preco" is a procedure with no MiDES modality, and becomes 98 (Processo licitatorio) rather than 99 (Outros/Nao aplicavel), which would assert no tender took place. - **indicador_restos_pagar is null.** nr_empenho carries the same year as `exercicio` in all 5,418,514 source rows, so a prior-year empenho is not identifiable. Checked rather than assumed. - **nome_credor / documento_credor on pagamento are joined from the empenho**, which is the only TO table carrying a creditor; supplier identity is one of the things MiDES exists to expose. Withheld where an empenho names more than one creditor across its lines, so a payment is never attributed to the wrong supplier. liquidacao's `credor` is left out of nome_responsavel on purpose: that column means the official who certified the liquidacao, not the supplier. Verified against the raw source -- every table reconciles to the cent: empenho 4,958,414 rows sum(valor_final) 46,822,295,283.38 liquidacao 7,930,599 rows 52,527,362,902.67 pagamento 8,385,628 rows 45,132,810,266.83 each equal to raw sum('+') - sum('-'), across 139/139 municipalities and 2013-2022. All 139 municipality codes resolve against br_bd_diretorios_brasil.municipio with no orphans, so the schema.yml FK tests hold. Parent column contracts are unchanged (25/20/25). Known imprecision, inherent to the source rather than the model: id_liquidacao_bd is withheld on 14.6% of rows and id_pagamento_bd on 19.1%, because one document number covers several budget lines and the key is genuinely ambiguous. Two liquidacao rows out of 7.9M have an unparseable date and land with a null date. --- .../states/world_wb_mides__empenho_to.sql | 178 ++++++++++++++++++ .../states/world_wb_mides__liquidacao_to.sql | 128 +++++++++++++ .../states/world_wb_mides__pagamento_to.sql | 173 +++++++++++++++++ .../world_wb_mides__empenho.sql | 3 + .../world_wb_mides__liquidacao.sql | 3 + .../world_wb_mides__pagamento.sql | 3 + 6 files changed, 488 insertions(+) create mode 100644 models/world_wb_mides/states/world_wb_mides__empenho_to.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__liquidacao_to.sql create mode 100644 models/world_wb_mides/states/world_wb_mides__pagamento_to.sql diff --git a/models/world_wb_mides/states/world_wb_mides__empenho_to.sql b/models/world_wb_mides/states/world_wb_mides__empenho_to.sql new file mode 100644 index 0000000000..9079b97dcc --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__empenho_to.sql @@ -0,0 +1,178 @@ +-- Tocantins (TO) contribution to world_wb_mides.empenho. +-- +-- Source: world_wb_mides_staging.raw_empenho_to, 2013-2022, all 139 TO +-- municipalities. TCE-TO publishes a *movement ledger*: one row per movement on +-- an empenho, signed by `sinal` ('+' issue, '-' anulacao). The MiDES empenho +-- schema wants one row per empenho line with the movements folded into +-- valor_inicial / valor_anulacao / valor_final, so the movements are aggregated +-- to the line key (municipio, orgao, nr_empenho, rubrica). +-- +-- The line key is the right grain, not (municipio, nr_empenho): of the 238,192 +-- empenho numbers carrying more than one '+' row, 228,729 (96%) differ in +-- `rubrica`, i.e. they are separate budget lines of one empenho rather than +-- reforcos. Aggregating on the empenho number alone would merge distinct lines +-- and attribute each line's anulacao to all of them. +-- +-- `indicador_restos_pagar` is null by necessity: nr_empenho always carries the +-- same year as `exercicio` in every one of the 5,418,514 source rows, so a +-- prior-year empenho is not identifiable here. +with + movimento_to as ( + select + safe_cast(exercicio as int64) as ano, + parse_date('%d/%m/%Y', trim(data)) as data, + safe_cast(municipio as string) as id_municipio, + safe_cast(trim(orgao) as string) as orgao, + safe_cast( + trim(split(unidade_gestora, ' - ')[safe_offset(0)]) as string + ) as id_unidade_gestora, + safe_cast(nr_empenho as string) as numero, + safe_cast(trim(rubrica) as string) as rubrica, + safe_cast(lower(historico) as string) as descricao, + safe_cast(safe_cast(funcao as int64) as string) as funcao, + safe_cast(safe_cast(subfuncao as int64) as string) as subfuncao, + safe_cast(safe_cast(programa as int64) as string) as programa, + safe_cast(safe_cast(proj_atividade as int64) as string) as acao, + -- TCE-TO's own modalidade codes do NOT match the MiDES dictionary + -- (TO 1 = Dispensa, MiDES 1 = Convite), so they are remapped by + -- description onto the codes in world_wb_mides.dicionario. TO's + -- "Registro de Preco" is a procurement procedure, not a modality, + -- and has no MiDES equivalent: it becomes 98 (Processo licitatorio), + -- the generic "a tender happened" code, rather than 99 (Outros/Nao + -- aplicavel), which would assert no tender took place. Where TO + -- qualifies a pregao as "- Registro de Preco", the pregao modality + -- is kept and the procedure dropped. + case + trim(modalidade_licitacao) + when '1' + then '8' -- Dispensa + when '2' + then '10' -- Inexigibilidade + when '3' + then '1' -- Convite + when '4' + then '2' -- Tomada de Precos + when '5' + then '3' -- Concorrencia + when '6' + then '98' -- Registro de Preco -> Processo licitatorio + when '7' + then '5' -- Pregao Presencial + when '8' + then '6' -- Pregao Eletronico + when '9' + then '5' -- Pregao Presencial - Registro de Preco + when '10' + then '6' -- Pregao Eletronico - Registro de Preco + when '11' + then '12' -- RDC + when '12' + then '31' -- Chamamento/Credenciamento -> Chamada Publica + when '99' + then '99' -- Nao aplicado + end as modalidade_licitacao, + trim(sinal) as sinal, + safe_cast(valor as float64) as valor + from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_to") }} + ), + linha_to as ( + select + ano, + id_municipio, + orgao, + numero, + rubrica, + min(data) as data, + any_value(id_unidade_gestora) as id_unidade_gestora, + any_value(descricao) as descricao, + any_value(funcao) as funcao, + any_value(subfuncao) as subfuncao, + any_value(programa) as programa, + any_value(acao) as acao, + any_value(modalidade_licitacao) as modalidade_licitacao, + round(sum(if(sinal = '+', valor, 0)), 2) as valor_inicial, + round(sum(if(sinal = '-', valor, 0)), 2) as valor_anulacao + from movimento_to + group by ano, id_municipio, orgao, numero, rubrica + ), + empenhado_to as ( + select + ano, + extract(month from data) as mes, + data, + 'TO' as sigla_uf, + id_municipio, + orgao, + id_unidade_gestora, + safe_cast(null as string) as id_licitacao_bd, + safe_cast(null as string) as id_licitacao, + modalidade_licitacao, + safe_cast( + concat( + numero, + ' ', + orgao, + ' ', + id_municipio, + ' ', + right(cast(ano as string), 2) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + numero, + descricao, + safe_cast(null as string) as modalidade, + funcao, + subfuncao, + programa, + acao, + safe_cast(substr(rubrica, 1, 8) as string) as elemento_despesa, + valor_inicial, + valor_anulacao + from linha_to + ), + frequencia_to as ( + select id_empenho_bd, count(id_empenho_bd) as frequencia_id + from empenhado_to + group by 1 + ), + empenho_to as ( + select + e.ano, + e.mes, + e.data, + e.sigla_uf, + e.id_municipio, + e.orgao, + e.id_unidade_gestora, + e.id_licitacao_bd, + e.id_licitacao, + e.modalidade_licitacao, + -- house convention: an empenho spanning several budget lines has an + -- ambiguous key, so it is withheld rather than published wrong. + ( + case + when f.frequencia_id > 1 + then safe_cast(null as string) + else e.id_empenho_bd + end + ) as id_empenho_bd, + e.id_empenho, + e.numero, + e.descricao, + e.modalidade, + e.funcao, + e.subfuncao, + e.programa, + e.acao, + e.elemento_despesa, + round(e.valor_inicial, 2) as valor_inicial, + round(safe_cast(0 as float64), 2) as valor_reforco, + round(e.valor_anulacao, 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(e.valor_inicial - e.valor_anulacao, 2) as valor_final + from empenhado_to e + left join frequencia_to f on e.id_empenho_bd = f.id_empenho_bd + ) +select * +from empenho_to diff --git a/models/world_wb_mides/states/world_wb_mides__liquidacao_to.sql b/models/world_wb_mides/states/world_wb_mides__liquidacao_to.sql new file mode 100644 index 0000000000..75d6105baf --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__liquidacao_to.sql @@ -0,0 +1,128 @@ +-- Tocantins (TO) contribution to world_wb_mides.liquidacao. +-- +-- Source: world_wb_mides_staging.raw_liquidacao_to, 2013-2022. Same movement +-- ledger shape as the TO empenho source: one row per movement, signed by +-- `sinal`, aggregated here to the line key +-- (municipio, orgao, nr_liquidacao, nr_empenho, rubrica). +-- +-- nome_responsavel / documento_responsavel are null on purpose. TCE-TO does +-- publish `credor` / `nome_credor` on this table, but that is the supplier +-- being paid, not the official who certified the liquidacao; putting a creditor +-- in a responsavel column would misstate what it means. The creditor is carried +-- on the empenho and on pagamento instead. +-- +-- indicador_restos_pagar is null for the reason given in the empenho model: +-- nr_empenho always carries the same year as `exercicio`, so an empenho +-- inherited from a previous year cannot be identified in this source. +with + movimento_to as ( + select + safe_cast(exercicio as int64) as ano, + parse_date('%d/%m/%Y', trim(data)) as data, + safe_cast(municipio as string) as id_municipio, + safe_cast(trim(orgao) as string) as orgao, + safe_cast( + trim(split(unidade_gestora, ' - ')[safe_offset(0)]) as string + ) as id_unidade_gestora, + safe_cast(nr_empenho as string) as numero_empenho, + safe_cast(nr_liquidacao as string) as numero, + safe_cast(trim(rubrica) as string) as rubrica, + trim(sinal) as sinal, + safe_cast(valor as float64) as valor + from {{ set_datalake_project("world_wb_mides_staging.raw_liquidacao_to") }} + ), + linha_to as ( + select + ano, + id_municipio, + orgao, + numero, + numero_empenho, + rubrica, + min(data) as data, + any_value(id_unidade_gestora) as id_unidade_gestora, + round(sum(if(sinal = '+', valor, 0)), 2) as valor_inicial, + round(sum(if(sinal = '-', valor, 0)), 2) as valor_anulacao + from movimento_to + group by ano, id_municipio, orgao, numero, numero_empenho, rubrica + ), + liquidado_to as ( + select + ano, + extract(month from data) as mes, + data, + 'TO' as sigla_uf, + id_municipio, + orgao, + id_unidade_gestora, + safe_cast( + concat( + numero_empenho, + ' ', + orgao, + ' ', + id_municipio, + ' ', + right(cast(ano as string), 2) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + numero_empenho, + safe_cast( + concat( + numero, + ' ', + orgao, + ' ', + id_municipio, + ' ', + right(cast(ano as string), 2) + ) as string + ) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + numero, + safe_cast(null as string) as nome_responsavel, + safe_cast(null as string) as documento_responsavel, + safe_cast(null as bool) as indicador_restos_pagar, + valor_inicial, + valor_anulacao + from linha_to + ), + frequencia_to as ( + select id_liquidacao_bd, count(id_liquidacao_bd) as frequencia_id + from liquidado_to + group by 1 + ), + liquidacao_to as ( + select + l.ano, + l.mes, + l.data, + l.sigla_uf, + l.id_municipio, + l.orgao, + l.id_unidade_gestora, + l.id_empenho_bd, + l.id_empenho, + l.numero_empenho, + ( + case + when f.frequencia_id > 1 + then safe_cast(null as string) + else l.id_liquidacao_bd + end + ) as id_liquidacao_bd, + l.id_liquidacao, + l.numero, + l.nome_responsavel, + l.documento_responsavel, + l.indicador_restos_pagar, + round(l.valor_inicial, 2) as valor_inicial, + round(l.valor_anulacao, 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(l.valor_inicial - l.valor_anulacao, 2) as valor_final + from liquidado_to l + left join frequencia_to f on l.id_liquidacao_bd = f.id_liquidacao_bd + ) +select * +from liquidacao_to diff --git a/models/world_wb_mides/states/world_wb_mides__pagamento_to.sql b/models/world_wb_mides/states/world_wb_mides__pagamento_to.sql new file mode 100644 index 0000000000..e6467eff38 --- /dev/null +++ b/models/world_wb_mides/states/world_wb_mides__pagamento_to.sql @@ -0,0 +1,173 @@ +-- Tocantins (TO) contribution to world_wb_mides.pagamento. +-- +-- Source: world_wb_mides_staging.raw_pagamento_to, 2013-2022. Same movement +-- ledger shape as the other two TO sources: one row per movement, signed by +-- `sinal`, aggregated here to the line key +-- (municipio, orgao, nr_pagamento, nr_liquidacao, nr_empenho, rubrica). +-- +-- TCE-TO's pagamento file carries no creditor, so nome_credor / documento_credor +-- are joined from the empenho on (municipio, nr_empenho) -- supplier identity is +-- one of the things MiDES exists to expose. The join is deliberately withheld +-- where an empenho names more than one creditor across its lines, so a payment +-- is never attributed to the wrong supplier. +-- +-- indicador_restos_pagar is null for the reason given in the empenho model. +-- valor_liquido_recebido is null: TCE-TO publishes the gross payment only, with +-- no net-of-deductions figure. +with + credor_to as ( + select + safe_cast(municipio as string) as id_municipio, + safe_cast(nr_empenho as string) as numero_empenho, + case + when count(distinct idcredor) = 1 + then any_value(safe_cast(nome_credor as string)) + end as nome_credor, + case + when count(distinct idcredor) = 1 + then any_value(safe_cast(idcredor as string)) + end as documento_credor + from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_to") }} + group by 1, 2 + ), + movimento_to as ( + select + safe_cast(exercicio as int64) as ano, + parse_date('%d/%m/%Y', trim(data)) as data, + safe_cast(municipio as string) as id_municipio, + safe_cast(trim(orgao) as string) as orgao, + safe_cast( + trim(split(unidade_gestora, ' - ')[safe_offset(0)]) as string + ) as id_unidade_gestora, + safe_cast(nr_empenho as string) as numero_empenho, + safe_cast(nr_liquidacao as string) as numero_liquidacao, + safe_cast(nr_pagamento as string) as numero, + safe_cast(trim(rubrica) as string) as rubrica, + safe_cast( + trim(split(rec_vinculado, ' - ')[safe_offset(0)]) as string + ) as fonte, + trim(sinal) as sinal, + safe_cast(valor as float64) as valor + from {{ set_datalake_project("world_wb_mides_staging.raw_pagamento_to") }} + ), + linha_to as ( + select + ano, + id_municipio, + orgao, + numero, + numero_liquidacao, + numero_empenho, + rubrica, + min(data) as data, + any_value(id_unidade_gestora) as id_unidade_gestora, + any_value(fonte) as fonte, + round(sum(if(sinal = '+', valor, 0)), 2) as valor_inicial, + round(sum(if(sinal = '-', valor, 0)), 2) as valor_anulacao + from movimento_to + group by + ano, id_municipio, orgao, numero, numero_liquidacao, numero_empenho, rubrica + ), + pago_to as ( + select + l.ano, + extract(month from l.data) as mes, + l.data, + 'TO' as sigla_uf, + l.id_municipio, + l.orgao, + l.id_unidade_gestora, + safe_cast( + concat( + l.numero_empenho, + ' ', + l.orgao, + ' ', + l.id_municipio, + ' ', + right(cast(l.ano as string), 2) + ) as string + ) as id_empenho_bd, + safe_cast(null as string) as id_empenho, + l.numero_empenho, + safe_cast( + concat( + l.numero_liquidacao, + ' ', + l.orgao, + ' ', + l.id_municipio, + ' ', + right(cast(l.ano as string), 2) + ) as string + ) as id_liquidacao_bd, + safe_cast(null as string) as id_liquidacao, + l.numero_liquidacao, + safe_cast( + concat( + l.numero, + ' ', + l.orgao, + ' ', + l.id_municipio, + ' ', + right(cast(l.ano as string), 2) + ) as string + ) as id_pagamento_bd, + safe_cast(null as string) as id_pagamento, + l.numero, + c.nome_credor, + c.documento_credor, + safe_cast(null as bool) as indicador_restos_pagar, + l.fonte, + l.valor_inicial, + l.valor_anulacao + from linha_to l + left join + credor_to c + on l.id_municipio = c.id_municipio + and l.numero_empenho = c.numero_empenho + ), + frequencia_to as ( + select id_pagamento_bd, count(id_pagamento_bd) as frequencia_id + from pago_to + group by 1 + ), + pagamento_to as ( + select + p.ano, + p.mes, + p.data, + p.sigla_uf, + p.id_municipio, + p.orgao, + p.id_unidade_gestora, + p.id_empenho_bd, + p.id_empenho, + p.numero_empenho, + p.id_liquidacao_bd, + p.id_liquidacao, + p.numero_liquidacao, + ( + case + when f.frequencia_id > 1 + then safe_cast(null as string) + else p.id_pagamento_bd + end + ) as id_pagamento_bd, + p.id_pagamento, + p.numero, + p.nome_credor, + p.documento_credor, + p.indicador_restos_pagar, + p.fonte, + round(p.valor_inicial, 2) as valor_inicial, + round(p.valor_anulacao, 2) as valor_anulacao, + round(safe_cast(0 as float64), 2) as valor_ajuste, + round(p.valor_inicial - p.valor_anulacao, 2) as valor_final, + safe_cast(null as float64) as valor_liquido_recebido + from pago_to p + left join frequencia_to f on p.id_pagamento_bd = f.id_pagamento_bd + ) +select * +from pagamento_to diff --git a/models/world_wb_mides/world_wb_mides__empenho.sql b/models/world_wb_mides/world_wb_mides__empenho.sql index c9054ad05a..335b61d694 100644 --- a/models/world_wb_mides/world_wb_mides__empenho.sql +++ b/models/world_wb_mides/world_wb_mides__empenho.sql @@ -69,4 +69,7 @@ from union all select * from {{ ref("world_wb_mides__empenho_sc") }} + union all + select * + from {{ ref("world_wb_mides__empenho_to") }} ) diff --git a/models/world_wb_mides/world_wb_mides__liquidacao.sql b/models/world_wb_mides/world_wb_mides__liquidacao.sql index fc4efa581c..8f4acfa084 100644 --- a/models/world_wb_mides/world_wb_mides__liquidacao.sql +++ b/models/world_wb_mides/world_wb_mides__liquidacao.sql @@ -64,4 +64,7 @@ from union all select * from {{ ref("world_wb_mides__liquidacao_sc") }} + union all + select * + from {{ ref("world_wb_mides__liquidacao_to") }} ) diff --git a/models/world_wb_mides/world_wb_mides__pagamento.sql b/models/world_wb_mides/world_wb_mides__pagamento.sql index 0ec3aed236..2a1b5e6e8f 100644 --- a/models/world_wb_mides/world_wb_mides__pagamento.sql +++ b/models/world_wb_mides/world_wb_mides__pagamento.sql @@ -69,4 +69,7 @@ from union all select * from {{ ref("world_wb_mides__pagamento_sc") }} + union all + select * + from {{ ref("world_wb_mides__pagamento_to") }} ) From 0244a38b9b1998749bbb7e94b26df2723ba5e095 Mon Sep 17 00:00:00 2001 From: Ricardo Dahis Date: Fri, 28 Aug 2026 12:23:41 +1000 Subject: [PATCH 5/6] Revert "feat(world_wb_mides): add Tocantins to empenho, liquidacao and pagamento" This reverts commit b3862c0c3. Backed out at the author's request: adding TO required several judgement calls about how TCE-TO's data maps onto the MiDES schema, and those need a domain review that is not available right now. Shipping them unverified would put choices nobody has checked into a published table. The calls in question, for whoever picks this up: - The source is a movement ledger (one row per movement, signed by `sinal`), aggregated to a line key of (municipio, orgao, numero, ..., rubrica) rather than the document number alone, because 96% of multi-row empenho numbers differ in `rubrica`. - modalidade_licitacao was remapped by description onto the MiDES dictionary, since TO's own codes disagree with it (TO 1 = Dispensa, MiDES 1 = Convite). TO's "Registro de Preco" has no MiDES modality and was mapped to 98. - nome_credor / documento_credor on pagamento were joined from the empenho, TO's only table carrying a creditor. - indicador_restos_pagar was left null because nr_empenho always carries the same year as `exercicio` in this source. The work reconciled to the cent against the raw source (empenho 4,958,414 rows / 46,822,295,283.38; liquidacao 7,930,599 / 52,527,362,902.67; pagamento 8,385,628 / 45,132,810,266.83, each equal to raw sum('+') - sum('-'), 139/139 municipalities FK-clean), so it is recoverable from b3862c0c3 once the mapping decisions have been reviewed. The rest of the branch is unaffected: the model cleanup, the per-state split and the table-approve depth fix all stand, and the three published tables still carry exactly the ten states they did before. --- .../states/world_wb_mides__empenho_to.sql | 178 ------------------ .../states/world_wb_mides__liquidacao_to.sql | 128 ------------- .../states/world_wb_mides__pagamento_to.sql | 173 ----------------- .../world_wb_mides__empenho.sql | 3 - .../world_wb_mides__liquidacao.sql | 3 - .../world_wb_mides__pagamento.sql | 3 - 6 files changed, 488 deletions(-) delete mode 100644 models/world_wb_mides/states/world_wb_mides__empenho_to.sql delete mode 100644 models/world_wb_mides/states/world_wb_mides__liquidacao_to.sql delete mode 100644 models/world_wb_mides/states/world_wb_mides__pagamento_to.sql diff --git a/models/world_wb_mides/states/world_wb_mides__empenho_to.sql b/models/world_wb_mides/states/world_wb_mides__empenho_to.sql deleted file mode 100644 index 9079b97dcc..0000000000 --- a/models/world_wb_mides/states/world_wb_mides__empenho_to.sql +++ /dev/null @@ -1,178 +0,0 @@ --- Tocantins (TO) contribution to world_wb_mides.empenho. --- --- Source: world_wb_mides_staging.raw_empenho_to, 2013-2022, all 139 TO --- municipalities. TCE-TO publishes a *movement ledger*: one row per movement on --- an empenho, signed by `sinal` ('+' issue, '-' anulacao). The MiDES empenho --- schema wants one row per empenho line with the movements folded into --- valor_inicial / valor_anulacao / valor_final, so the movements are aggregated --- to the line key (municipio, orgao, nr_empenho, rubrica). --- --- The line key is the right grain, not (municipio, nr_empenho): of the 238,192 --- empenho numbers carrying more than one '+' row, 228,729 (96%) differ in --- `rubrica`, i.e. they are separate budget lines of one empenho rather than --- reforcos. Aggregating on the empenho number alone would merge distinct lines --- and attribute each line's anulacao to all of them. --- --- `indicador_restos_pagar` is null by necessity: nr_empenho always carries the --- same year as `exercicio` in every one of the 5,418,514 source rows, so a --- prior-year empenho is not identifiable here. -with - movimento_to as ( - select - safe_cast(exercicio as int64) as ano, - parse_date('%d/%m/%Y', trim(data)) as data, - safe_cast(municipio as string) as id_municipio, - safe_cast(trim(orgao) as string) as orgao, - safe_cast( - trim(split(unidade_gestora, ' - ')[safe_offset(0)]) as string - ) as id_unidade_gestora, - safe_cast(nr_empenho as string) as numero, - safe_cast(trim(rubrica) as string) as rubrica, - safe_cast(lower(historico) as string) as descricao, - safe_cast(safe_cast(funcao as int64) as string) as funcao, - safe_cast(safe_cast(subfuncao as int64) as string) as subfuncao, - safe_cast(safe_cast(programa as int64) as string) as programa, - safe_cast(safe_cast(proj_atividade as int64) as string) as acao, - -- TCE-TO's own modalidade codes do NOT match the MiDES dictionary - -- (TO 1 = Dispensa, MiDES 1 = Convite), so they are remapped by - -- description onto the codes in world_wb_mides.dicionario. TO's - -- "Registro de Preco" is a procurement procedure, not a modality, - -- and has no MiDES equivalent: it becomes 98 (Processo licitatorio), - -- the generic "a tender happened" code, rather than 99 (Outros/Nao - -- aplicavel), which would assert no tender took place. Where TO - -- qualifies a pregao as "- Registro de Preco", the pregao modality - -- is kept and the procedure dropped. - case - trim(modalidade_licitacao) - when '1' - then '8' -- Dispensa - when '2' - then '10' -- Inexigibilidade - when '3' - then '1' -- Convite - when '4' - then '2' -- Tomada de Precos - when '5' - then '3' -- Concorrencia - when '6' - then '98' -- Registro de Preco -> Processo licitatorio - when '7' - then '5' -- Pregao Presencial - when '8' - then '6' -- Pregao Eletronico - when '9' - then '5' -- Pregao Presencial - Registro de Preco - when '10' - then '6' -- Pregao Eletronico - Registro de Preco - when '11' - then '12' -- RDC - when '12' - then '31' -- Chamamento/Credenciamento -> Chamada Publica - when '99' - then '99' -- Nao aplicado - end as modalidade_licitacao, - trim(sinal) as sinal, - safe_cast(valor as float64) as valor - from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_to") }} - ), - linha_to as ( - select - ano, - id_municipio, - orgao, - numero, - rubrica, - min(data) as data, - any_value(id_unidade_gestora) as id_unidade_gestora, - any_value(descricao) as descricao, - any_value(funcao) as funcao, - any_value(subfuncao) as subfuncao, - any_value(programa) as programa, - any_value(acao) as acao, - any_value(modalidade_licitacao) as modalidade_licitacao, - round(sum(if(sinal = '+', valor, 0)), 2) as valor_inicial, - round(sum(if(sinal = '-', valor, 0)), 2) as valor_anulacao - from movimento_to - group by ano, id_municipio, orgao, numero, rubrica - ), - empenhado_to as ( - select - ano, - extract(month from data) as mes, - data, - 'TO' as sigla_uf, - id_municipio, - orgao, - id_unidade_gestora, - safe_cast(null as string) as id_licitacao_bd, - safe_cast(null as string) as id_licitacao, - modalidade_licitacao, - safe_cast( - concat( - numero, - ' ', - orgao, - ' ', - id_municipio, - ' ', - right(cast(ano as string), 2) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - numero, - descricao, - safe_cast(null as string) as modalidade, - funcao, - subfuncao, - programa, - acao, - safe_cast(substr(rubrica, 1, 8) as string) as elemento_despesa, - valor_inicial, - valor_anulacao - from linha_to - ), - frequencia_to as ( - select id_empenho_bd, count(id_empenho_bd) as frequencia_id - from empenhado_to - group by 1 - ), - empenho_to as ( - select - e.ano, - e.mes, - e.data, - e.sigla_uf, - e.id_municipio, - e.orgao, - e.id_unidade_gestora, - e.id_licitacao_bd, - e.id_licitacao, - e.modalidade_licitacao, - -- house convention: an empenho spanning several budget lines has an - -- ambiguous key, so it is withheld rather than published wrong. - ( - case - when f.frequencia_id > 1 - then safe_cast(null as string) - else e.id_empenho_bd - end - ) as id_empenho_bd, - e.id_empenho, - e.numero, - e.descricao, - e.modalidade, - e.funcao, - e.subfuncao, - e.programa, - e.acao, - e.elemento_despesa, - round(e.valor_inicial, 2) as valor_inicial, - round(safe_cast(0 as float64), 2) as valor_reforco, - round(e.valor_anulacao, 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(e.valor_inicial - e.valor_anulacao, 2) as valor_final - from empenhado_to e - left join frequencia_to f on e.id_empenho_bd = f.id_empenho_bd - ) -select * -from empenho_to diff --git a/models/world_wb_mides/states/world_wb_mides__liquidacao_to.sql b/models/world_wb_mides/states/world_wb_mides__liquidacao_to.sql deleted file mode 100644 index 75d6105baf..0000000000 --- a/models/world_wb_mides/states/world_wb_mides__liquidacao_to.sql +++ /dev/null @@ -1,128 +0,0 @@ --- Tocantins (TO) contribution to world_wb_mides.liquidacao. --- --- Source: world_wb_mides_staging.raw_liquidacao_to, 2013-2022. Same movement --- ledger shape as the TO empenho source: one row per movement, signed by --- `sinal`, aggregated here to the line key --- (municipio, orgao, nr_liquidacao, nr_empenho, rubrica). --- --- nome_responsavel / documento_responsavel are null on purpose. TCE-TO does --- publish `credor` / `nome_credor` on this table, but that is the supplier --- being paid, not the official who certified the liquidacao; putting a creditor --- in a responsavel column would misstate what it means. The creditor is carried --- on the empenho and on pagamento instead. --- --- indicador_restos_pagar is null for the reason given in the empenho model: --- nr_empenho always carries the same year as `exercicio`, so an empenho --- inherited from a previous year cannot be identified in this source. -with - movimento_to as ( - select - safe_cast(exercicio as int64) as ano, - parse_date('%d/%m/%Y', trim(data)) as data, - safe_cast(municipio as string) as id_municipio, - safe_cast(trim(orgao) as string) as orgao, - safe_cast( - trim(split(unidade_gestora, ' - ')[safe_offset(0)]) as string - ) as id_unidade_gestora, - safe_cast(nr_empenho as string) as numero_empenho, - safe_cast(nr_liquidacao as string) as numero, - safe_cast(trim(rubrica) as string) as rubrica, - trim(sinal) as sinal, - safe_cast(valor as float64) as valor - from {{ set_datalake_project("world_wb_mides_staging.raw_liquidacao_to") }} - ), - linha_to as ( - select - ano, - id_municipio, - orgao, - numero, - numero_empenho, - rubrica, - min(data) as data, - any_value(id_unidade_gestora) as id_unidade_gestora, - round(sum(if(sinal = '+', valor, 0)), 2) as valor_inicial, - round(sum(if(sinal = '-', valor, 0)), 2) as valor_anulacao - from movimento_to - group by ano, id_municipio, orgao, numero, numero_empenho, rubrica - ), - liquidado_to as ( - select - ano, - extract(month from data) as mes, - data, - 'TO' as sigla_uf, - id_municipio, - orgao, - id_unidade_gestora, - safe_cast( - concat( - numero_empenho, - ' ', - orgao, - ' ', - id_municipio, - ' ', - right(cast(ano as string), 2) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - numero_empenho, - safe_cast( - concat( - numero, - ' ', - orgao, - ' ', - id_municipio, - ' ', - right(cast(ano as string), 2) - ) as string - ) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - numero, - safe_cast(null as string) as nome_responsavel, - safe_cast(null as string) as documento_responsavel, - safe_cast(null as bool) as indicador_restos_pagar, - valor_inicial, - valor_anulacao - from linha_to - ), - frequencia_to as ( - select id_liquidacao_bd, count(id_liquidacao_bd) as frequencia_id - from liquidado_to - group by 1 - ), - liquidacao_to as ( - select - l.ano, - l.mes, - l.data, - l.sigla_uf, - l.id_municipio, - l.orgao, - l.id_unidade_gestora, - l.id_empenho_bd, - l.id_empenho, - l.numero_empenho, - ( - case - when f.frequencia_id > 1 - then safe_cast(null as string) - else l.id_liquidacao_bd - end - ) as id_liquidacao_bd, - l.id_liquidacao, - l.numero, - l.nome_responsavel, - l.documento_responsavel, - l.indicador_restos_pagar, - round(l.valor_inicial, 2) as valor_inicial, - round(l.valor_anulacao, 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(l.valor_inicial - l.valor_anulacao, 2) as valor_final - from liquidado_to l - left join frequencia_to f on l.id_liquidacao_bd = f.id_liquidacao_bd - ) -select * -from liquidacao_to diff --git a/models/world_wb_mides/states/world_wb_mides__pagamento_to.sql b/models/world_wb_mides/states/world_wb_mides__pagamento_to.sql deleted file mode 100644 index e6467eff38..0000000000 --- a/models/world_wb_mides/states/world_wb_mides__pagamento_to.sql +++ /dev/null @@ -1,173 +0,0 @@ --- Tocantins (TO) contribution to world_wb_mides.pagamento. --- --- Source: world_wb_mides_staging.raw_pagamento_to, 2013-2022. Same movement --- ledger shape as the other two TO sources: one row per movement, signed by --- `sinal`, aggregated here to the line key --- (municipio, orgao, nr_pagamento, nr_liquidacao, nr_empenho, rubrica). --- --- TCE-TO's pagamento file carries no creditor, so nome_credor / documento_credor --- are joined from the empenho on (municipio, nr_empenho) -- supplier identity is --- one of the things MiDES exists to expose. The join is deliberately withheld --- where an empenho names more than one creditor across its lines, so a payment --- is never attributed to the wrong supplier. --- --- indicador_restos_pagar is null for the reason given in the empenho model. --- valor_liquido_recebido is null: TCE-TO publishes the gross payment only, with --- no net-of-deductions figure. -with - credor_to as ( - select - safe_cast(municipio as string) as id_municipio, - safe_cast(nr_empenho as string) as numero_empenho, - case - when count(distinct idcredor) = 1 - then any_value(safe_cast(nome_credor as string)) - end as nome_credor, - case - when count(distinct idcredor) = 1 - then any_value(safe_cast(idcredor as string)) - end as documento_credor - from {{ set_datalake_project("world_wb_mides_staging.raw_empenho_to") }} - group by 1, 2 - ), - movimento_to as ( - select - safe_cast(exercicio as int64) as ano, - parse_date('%d/%m/%Y', trim(data)) as data, - safe_cast(municipio as string) as id_municipio, - safe_cast(trim(orgao) as string) as orgao, - safe_cast( - trim(split(unidade_gestora, ' - ')[safe_offset(0)]) as string - ) as id_unidade_gestora, - safe_cast(nr_empenho as string) as numero_empenho, - safe_cast(nr_liquidacao as string) as numero_liquidacao, - safe_cast(nr_pagamento as string) as numero, - safe_cast(trim(rubrica) as string) as rubrica, - safe_cast( - trim(split(rec_vinculado, ' - ')[safe_offset(0)]) as string - ) as fonte, - trim(sinal) as sinal, - safe_cast(valor as float64) as valor - from {{ set_datalake_project("world_wb_mides_staging.raw_pagamento_to") }} - ), - linha_to as ( - select - ano, - id_municipio, - orgao, - numero, - numero_liquidacao, - numero_empenho, - rubrica, - min(data) as data, - any_value(id_unidade_gestora) as id_unidade_gestora, - any_value(fonte) as fonte, - round(sum(if(sinal = '+', valor, 0)), 2) as valor_inicial, - round(sum(if(sinal = '-', valor, 0)), 2) as valor_anulacao - from movimento_to - group by - ano, id_municipio, orgao, numero, numero_liquidacao, numero_empenho, rubrica - ), - pago_to as ( - select - l.ano, - extract(month from l.data) as mes, - l.data, - 'TO' as sigla_uf, - l.id_municipio, - l.orgao, - l.id_unidade_gestora, - safe_cast( - concat( - l.numero_empenho, - ' ', - l.orgao, - ' ', - l.id_municipio, - ' ', - right(cast(l.ano as string), 2) - ) as string - ) as id_empenho_bd, - safe_cast(null as string) as id_empenho, - l.numero_empenho, - safe_cast( - concat( - l.numero_liquidacao, - ' ', - l.orgao, - ' ', - l.id_municipio, - ' ', - right(cast(l.ano as string), 2) - ) as string - ) as id_liquidacao_bd, - safe_cast(null as string) as id_liquidacao, - l.numero_liquidacao, - safe_cast( - concat( - l.numero, - ' ', - l.orgao, - ' ', - l.id_municipio, - ' ', - right(cast(l.ano as string), 2) - ) as string - ) as id_pagamento_bd, - safe_cast(null as string) as id_pagamento, - l.numero, - c.nome_credor, - c.documento_credor, - safe_cast(null as bool) as indicador_restos_pagar, - l.fonte, - l.valor_inicial, - l.valor_anulacao - from linha_to l - left join - credor_to c - on l.id_municipio = c.id_municipio - and l.numero_empenho = c.numero_empenho - ), - frequencia_to as ( - select id_pagamento_bd, count(id_pagamento_bd) as frequencia_id - from pago_to - group by 1 - ), - pagamento_to as ( - select - p.ano, - p.mes, - p.data, - p.sigla_uf, - p.id_municipio, - p.orgao, - p.id_unidade_gestora, - p.id_empenho_bd, - p.id_empenho, - p.numero_empenho, - p.id_liquidacao_bd, - p.id_liquidacao, - p.numero_liquidacao, - ( - case - when f.frequencia_id > 1 - then safe_cast(null as string) - else p.id_pagamento_bd - end - ) as id_pagamento_bd, - p.id_pagamento, - p.numero, - p.nome_credor, - p.documento_credor, - p.indicador_restos_pagar, - p.fonte, - round(p.valor_inicial, 2) as valor_inicial, - round(p.valor_anulacao, 2) as valor_anulacao, - round(safe_cast(0 as float64), 2) as valor_ajuste, - round(p.valor_inicial - p.valor_anulacao, 2) as valor_final, - safe_cast(null as float64) as valor_liquido_recebido - from pago_to p - left join frequencia_to f on p.id_pagamento_bd = f.id_pagamento_bd - ) -select * -from pagamento_to diff --git a/models/world_wb_mides/world_wb_mides__empenho.sql b/models/world_wb_mides/world_wb_mides__empenho.sql index 335b61d694..c9054ad05a 100644 --- a/models/world_wb_mides/world_wb_mides__empenho.sql +++ b/models/world_wb_mides/world_wb_mides__empenho.sql @@ -69,7 +69,4 @@ from union all select * from {{ ref("world_wb_mides__empenho_sc") }} - union all - select * - from {{ ref("world_wb_mides__empenho_to") }} ) diff --git a/models/world_wb_mides/world_wb_mides__liquidacao.sql b/models/world_wb_mides/world_wb_mides__liquidacao.sql index 8f4acfa084..fc4efa581c 100644 --- a/models/world_wb_mides/world_wb_mides__liquidacao.sql +++ b/models/world_wb_mides/world_wb_mides__liquidacao.sql @@ -64,7 +64,4 @@ from union all select * from {{ ref("world_wb_mides__liquidacao_sc") }} - union all - select * - from {{ ref("world_wb_mides__liquidacao_to") }} ) diff --git a/models/world_wb_mides/world_wb_mides__pagamento.sql b/models/world_wb_mides/world_wb_mides__pagamento.sql index 2a1b5e6e8f..0ec3aed236 100644 --- a/models/world_wb_mides/world_wb_mides__pagamento.sql +++ b/models/world_wb_mides/world_wb_mides__pagamento.sql @@ -69,7 +69,4 @@ from union all select * from {{ ref("world_wb_mides__pagamento_sc") }} - union all - select * - from {{ ref("world_wb_mides__pagamento_to") }} ) From 9f9535932d37abdc00da6b3a1646fab126a13800 Mon Sep 17 00:00:00 2001 From: Ricardo Dahis Date: Fri, 18 Sep 2026 14:40:01 +1000 Subject: [PATCH 6/6] fix(world_wb_mides): RJ v2 municipal payments carried empenho ids in the payment id columns The `pagamento_municipio_rj_v2` select list emitted `p.id_empenho_bd` and `p.id_empenho` twice -- at positions 8-9, where they belong, and again at positions 14-15, where the published schema has `id_pagamento_bd` and `id_pagamento`. `union all` resolves positionally, so those two columns have been publishing empenho identifiers for every RJ v2 municipal row, and the `id_pagamento_bd` that `pago_municipio_rj_v2` actually builds was never emitted. Column counts matched, so the union compiled and the defect stayed invisible. Measured in the published table: **956,690 rows** of `basedosdados.world_wb_mides.pagamento` have `id_pagamento_bd = id_empenho_bd` -- 6.3% of RJ, and exactly the v2 municipal population (the remaining 14,118,437 RJ rows carry a null `id_pagamento_bd` from the v1 and state branches). This predates the per-state split: the same duplicated pair is in the monolithic `pagamento.sql` on main at lines 1372-1373 and 1378-1379. The split carried it over byte-identical, as intended, and CodeRabbit caught it on review. Swept all 30 state models for the same class of defect -- a union term whose output names diverge positionally from the parent's column list -- and this is the only genuine instance. The other flags were an alias-parsing artifact (`extract(month from date(...))` truncates a naive select-list parse) or columns that are positionally correct but unaliased. Not fixed here, because they are naming rather than data defects: `empenho_ce` and `empenho_municipio_rj_v1` both label position 20 `modalidade_despesa` where the table calls it `elemento_despesa`. The values are in the right column. Verified: sqlfmt passes, the dataset compiles, and all three parents dry-run with their contracts unchanged (25/20/25). Compiled `pagamento` positions 13-16 now read numero_liquidacao, id_pagamento_bd, id_pagamento, numero. Because the merge carries the `table-approve` label, the rebuild it triggers will correct these 956,690 rows rather than republish them. --- models/world_wb_mides/states/world_wb_mides__pagamento_rj.sql | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/models/world_wb_mides/states/world_wb_mides__pagamento_rj.sql b/models/world_wb_mides/states/world_wb_mides__pagamento_rj.sql index 1e8c36faa9..52a06cc559 100644 --- a/models/world_wb_mides/states/world_wb_mides__pagamento_rj.sql +++ b/models/world_wb_mides/states/world_wb_mides__pagamento_rj.sql @@ -223,8 +223,8 @@ with p.id_liquidacao_bd, p.id_liquidacao, p.numero_liquidacao, - p.id_empenho_bd, - p.id_empenho, + p.id_pagamento_bd, + p.id_pagamento, p.numero, p.nome_credor, p.documento_credor,