Cannot parse valid file

[alabamenhu]

I'm not sure at the moment where the parse failure is happening, but other validators confirm the file (from Unicode) is valid XML.

(rename file from ee.txt to ee.xml because GH doesn't like XML uploads for some reason).

I'll try to investigate further, but my guess is whatever is causing the problem is also causing problems with kn.xml, ks.xml, lo.xml, ml.xml, mr.xml, and yav.xml inside of the main/ directory for the CLDR repository.

ee.txt

[dwarring]

Here's a minimal failing example, snipped form the above :

<territory type="AG">́Antigua kple Barbuda nutome</territory>

The '>' is followed by a 0x301 COMBINING ACUTE ACCENT, giving us '>́' .

which is a pretty weird edge case.

[dwarring]

This document has a Unicode Normalization error - see XML 1.1 Specification section on Unicode Normalization.

[alabamenhu]

This document has a Unicode Normalization error - see XML 1.1 Speceification section on Unicode Normalization.

A Unicode-published doc with a Unicode normalization error? Why I'd never… ;-)

Knowing the source of the error, it should be easy for me to preprocess these files to avoid the issue. I'll report any ones that I find in the future to Unicode, but the text “should” doesn't require, so it seems like it's still a technically a valid XML file (even if no one should ever write it)

[dwarring]

There's also a differences in the decoding of this file.

Rakudo is working strictly at the UTF-8 character/code-point level and interpreting this as an accented closing tag '>́' .

LibXML (for example) is parsing at the grapheme level and interpreting this as a '>' followed by a stray COMBINING ACUTE ACCENT character.

[alabamenhu]

There's also a differences in the decoding of this file.

Rakudo is working strictly at the UTF-8 character/code-point level and interpreting this as an accented closing tag '>́' .

LibXML (for example) is parsing at the grapheme level and interpreting this as a '>' followed by a stray COMBINING ACUTE ACCENT character.

I think you flipped the two (Rakudo works on graphemes, most everyone else on codepoints), but this is the issue source. See the related JSON::Tiny issue.

The solution was to match (:ignoremark '"')" ~ \" <str> as the string and delimiters, and then in the actions, use

    if $0.Str ne '"' {
        my @chars := $0.Str.NFC;
        $str = @chars[1..*].chrs ~ $str;
    }
    make $str

Not an impossible fix to do, but given that stringy content occurs bare in between tags, it'll be a bit more than a C&P from JSON::Tiny

[dwarring]

Still a question of which is valid.

[alabamenhu]

Still a question of which is valid.

Looking over the XML standard, it's fairly clear that it's the way that LibXML does it. The grammar definitions provided use Char as the basis for most things, and Char is defined as

Char ::= [#x1-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] | /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */

In the definition for Character Data and Markup, we get:

In the content of elements, character data is any string of characters which does not contain the start-delimiter of any markup or the CDATA-section-close delimiter, "]]>". In a CDATA section, character data is any string of characters not including the CDATA-section-close delimiter.

Given that in the definition for Name, combining characters are explicitly disallowed as a StartChar (but not for the content), there could have been a formal restriction.

Nonetheless, the docs do state:

All XML parsed entities (including document entities) should be fully normalized as per the definition of B Definitions for Character Normalization supplemented by the following definitions of relevant constructs for XML:

• The replacement text of all parsed entities
• All text matching, in context, one of the following productions:
  • CData
  • CharData
  • content
  • Name
  • Nmtoken

However, a document is still well-formed even if it is not fully normalized.

[dwarring]

This file can be read, as is, with the aide of LibXML:

use LibXML::SAX;
use LibXML::SAX::Handler::XML;
use XML::Document;
my LibXML::SAX::Handler::XML $sax-handler .= new;
my LibXML::SAX $generator .= new: :$sax-handler;

my XML::Document $doc = $generator.parse: :file</tmp/ee.xml>, :$sax-handler;

This uses LibXML SAX to parse the file and construct an XML document on the file.

Probably a useful as a quick work-around when you can parse an XML document for any reason but have LibXML available.

update or in LibXML 0.5.6+:

use LibXML;
use LibXML::SAX::Handler::XML;
use XML::Document;

my LibXML::SAX::Handler::XML $sax-handler .= new;
my XML::Document $doc = LibXML.parse: :file</tmp/ee.xml>, :$sax-handler;

Note: this is currently about 3-4x faster that XML's built-in parser, so may have performance benefits as well.

[dwarring]

Agree it's a bug. We can easily create an XML file that can't be reparsed:

use XML;
use XML::Element;
use XML::Text;
use Test;
plan 1;

my XML::Element $e .= new: :name<Foo>;
my XML::Text $t .= new: :text("\c[COMBINING ACUTE ACCENT]ouch");

$e.insert($t);

lives-ok {from-xml($e.Str)};

Currently produces:

1..1
not ok 1 - 
# Failed test at /tmp/tstx.raku line 12
# could not parse XML
# You failed 1 test of 1

Because '>' + c[COMBINING ACUTE ACCENT] is normalising to '>́' .

[jonathanstowe]

Is this actually solvable?

[alabamenhu]

Is this actually solvable?

Yeah, JSON::Tiny can do it, so we should to. It's just a matter of finding every place where this can potentially happen and adjusting parsing accordingly.

It will, unfortunately, necessarily incur a performance penalty (needing to check for the composed character even though 99.99999% of the time, it won't be there).

I actually had forgotten about this, and I'll see if I can toss together a working solution + test files soon.

[jubilatious1]

Here's a minimal failing example, snipped form the above :

<territory type="AG">́Antigua kple Barbuda nutome</territory>

The '>' is followed by a 0x301 COMBINING ACUTE ACCENT, giving us '>́' .

which is a pretty weird edge case.

Confirmed on older MacOS with Rakudo_2021.06, LibXML:ver<0.6.4>:authcpan:WARRINGD;
"ee.xml" file parses using code/modules as in #59 (comment), with this line in output:

<territory type="AG">́Antigua kple Barbuda nutome</territory>

[JJ]

Is this solved? Or unsolvable?

[jubilatious1]

@samcv thoughts appreciated.

[alabamenhu]

Is this solved? Or unsolvable?

Not solved yet, but it is solvable. I'll see if I can get a PR in here in the next week or two -- it's really just a matter of C&Ping what JSON::Tiny does into XML. It will necessarily slow down the parsing a bit, but there's not much that can be down there given the way Raku handles Unicode. Then again, the baseline XML module is kinda slow anyways for other reasons, so probably not the end of the world.