case insensitive field matching behavior different from iceberg-python and iceberg-java

[nvartolomei]

#180 added support for GetFieldByName with case_sensitive argument. Its behavior is different from Python and Java Iceberg implementations which could result in inconsistent behavior when used together.

Python/Java are unicode/multibyte aware. iceberg-cpp is not. An example where this breaks e.g. column names containing e.g. "Latin Capital Letter I with Dot Above" https://www.compart.com/en/unicode/U+0130.

Does not affect me directly at this moment.

[wgtmac]

Thanks for reporting this issue. I agree that currently case sensitivity only supports ASCII characters. We might need to introduce https://github.com/unicode-org/icu to address this due to lack support of the C++ standard library.

[nvartolomei]

Using iceberg-cpp with glue rest catalog could produce incorrect results. Python script which tests Glue behavior test_column_names.py

============================================================
TEST: mixed_case
  table: glue_column_test.col_test_mixed_case_04236e66
  requested columns: ['userId', 'FirstName', 'last_NAME', 'isActive', 'XMLPayload']
  returned columns:  ['userid', 'firstname', 'last_name', 'isactive', 'xmlpayload']
  RESULT: MISMATCH
                          'userId' -> 'userid' <-- CHANGED
                       'FirstName' -> 'firstname' <-- CHANGED
                       'last_NAME' -> 'last_name' <-- CHANGED
                        'isActive' -> 'isactive' <-- CHANGED
                      'XMLPayload' -> 'xmlpayload' <-- CHANGED

============================================================
TEST: non_ascii
  table: glue_column_test.col_test_non_ascii_04236e66
  requested columns: ['naïve_count', 'café_id', '日本語カラム', 'Ціна', 'straße', 'İstanbul_İd', 'GROẞE_id']
  returned columns:  ['naïve_count', 'café_id', '日本語カラム', 'ціна', 'straße', 'i̇stanbul_i̇d', 'große_id']
  RESULT: MISMATCH
                     'naïve_count' -> 'naïve_count'
                         'café_id' -> 'café_id'
                          '日本語カラム' -> '日本語カラム'
                            'Ціна' -> 'ціна' <-- CHANGED
                          'straße' -> 'straße'
                     'İstanbul_İd' -> 'i̇stanbul_i̇d' <-- CHANGED
                        'GROẞE_id' -> 'große_id' <-- CHANGED

============================================================
TEST: mixed_case_emoji
  table: glue_column_test.col_test_mixed_case_emoji_04236e66
  requested columns: ['UserId🔑', '🔥HotMetric', 'Status_🟢_Active', 'café_Naïve_💀']
  returned columns:  ['userid🔑', '🔥hotmetric', 'status_🟢_active', 'café_naïve_💀']
  RESULT: MISMATCH
                         'UserId🔑' -> 'userid🔑' <-- CHANGED
                      '🔥HotMetric' -> '🔥hotmetric' <-- CHANGED
                 'Status_🟢_Active' -> 'status_🟢_active' <-- CHANGED
                    'café_Naïve_💀' -> 'café_naïve_💀' <-- CHANGED

============================================================
TEST: emoji
  table: glue_column_test.col_test_emoji_04236e66
  requested columns: ['🔥hot_metric', 'status_🟢', 'col_💀_danger', '🎉🎊🥳']
  returned columns:  ['🔥hot_metric', 'status_🟢', 'col_💀_danger', '🎉🎊🥳']
  RESULT: names round-tripped exactly

============================================================

[nvartolomei]

Ah. One more detail. AWS Glue does not always lowercase column names https://gist.github.com/nvartolomei/0a92ef2e0c8cc71ae87fe99d4b2c9acc