O noes, Unicode Normalization

While reading on some details Unicode normalization, I found a writeup of how URLs could be significantly changed by the process:

Unicode normalization could change the structure of a URL · Issue #626 · whatwg/url · GitHub

That's nuts.

To quickly refresh everyone's memory on our URL! datatype: we now think of it as being the decoded form...like what people would copy and paste out of a browser's URL bar. That could contain forms not legal to GET or POST to a web server. So the percent-encoding is something one has to choose to do with a string conversion to or from a URL! if applicable:

Treat URL! as normal strings with no encoding behavior by hostilefork · Pull Request #655 · metaeducation/ren-c · GitHub

However the GitHub issue I link up top points out that normalization is another nuance that is a place for bugs / hacking / etc. This jumped out to me as a very simple--yet insidious--thing:

NFD and NFKD will normalize the following three characters to generate potentially forbidden code points:

  • \u2260 (≠ as one code point) to =\u0338 (≠ as two code points)
  • \u226E (≮ as one code point) to <\u0338 (≮ as two code points)
  • \u226F (≯ as one code point) to >\u0338 (≯ as two code points)

So you aren't supposed to have < or > or = in the URL, but they can pop up as part of the unicode normalization process if you decompose things.

I've been pushing against automatic transformations as a broken concept--even when it was just carriage returns and line feeds.

Here I have the same instincts creeping in. So what do we know?

  • We should not be building a system that normalizes or denormalize anything without being asked.

    • We either accept only one format as input, or accept more formats and act neutrally.

    • I am a fan of strong stances over neutral ones, in order to guide the ecosystem to a better place, even if it makes interoperability with things outside the ecosystem require more effort

      • The "automatic" things that permit "interoperability" are a false economy

      • You pay with confusion, bugs, and security holes

  • The bias of stored formats on disks and filesystems is NFC

    • As canonized forms go it is not just the most compact, but it is also the most typical way to find codepoints encoded in practice.

    • It's what the W3C suggests be used for transfer over the Internet, regardless of the internal forms a program might use

    • Chrome is the leading browser by miles, and for é it canonizes both the one-codepoint (NFC) and separate e-and-accent form (NFD) to a network request for e.g. http://example.com/%C3%A9.txt. While other browsers like Firefox don't canonize, hence the two-codepoint form shows as http://example.com/e%CC%81.txt with the naked e and the accent codepoint two-byte UTF-8 percent encoded afterward.

      • Chrome's outsized influence suggests where the wind is blowing.
  • When you think about streaming, e.g. bytes flowing one by one from your program to something like the Windows or Linux terminal to display, decomposed forms give rise to chaos

    • Let's say you are the terminal program, and over the pipe you get something like the bytes for a prompt, like > followed by >.

      • Should you display those characters on the screen? What if a composable character comes afterward, to transform it into ≯

      • You might not have the power to cursor back and erase what you already printed, depending. But even if you can, that's a lot of complexity that has to be borne by every consumer of UTF-8 bytes

This is making Latin1 seem like a saner choice, and that engaging Unicode has actually been a mistake.

Is "NFC Everywhere" Possible?

We already enforce atomic manipulations to strings at the byte level so that you can only insert/remove byte sequences that leave the string as valid UTF-8.

The canonization process is standardized, and ostensibly we know what the combining characters are. Wouldn't be trivial to implement, but we could say all modifications must generate NFC.

>> string: "e"
== "e"

>> append string #{CC81}  ; the accent combining character in BINARY!
** Error: Modification to string would create non-canon form

That's different from today's answer, which lets you do it:

>> append string #{CC81}
== "é"

I'm not particularly afraid to go in this direction, but there are some glitches.

I gather not all combining character sequences people might use canonize to a single codepoint. Some meaningful sequences to readers are only available as their decomposed forms, and normalization just ensures that the combining marks are in the same byte order.

This means that if you're in the position of the terminal implementer, then so long as you are dealing with a character that could be composed with something in non-canonized form you'd have to wait for a non-composable character (?) before you print anything... or be able to go back. It seems to me nonsensical that the combining characters come after the character you combine with. Maybe we should buck the trend with our own format that puts them first, at least in streaming cases. :slight_smile:

Anyway, just wanted to write down some more thoughts about this heinous thing.