# O noes, Unicode Normalization

**URL:** <https://revolt.metaeducation.com/forum/t/o-noes-unicode-normalization/1322>\
**Category:** Internals\
**Created:** [August 27, 2020, 4:08pm UTC](https://revolt.metaeducation.com/forum/t/o-noes-unicode-normalization/1322 "2020-08-27T16:08:09Z")\
**Posts on this page:** 1\
**Showing post:** 9

<div class="post-metadata">

**Author:** ![hostilefork](https://revolt.metaeducation.com/forum/user_avatar/revolt.metaeducation.com/hostilefork/32/421_2.png) [@hostilefork](https://revolt.metaeducation.com/forum/u/hostilefork)\
**Post date:** [June 12, 2024, 11:26pm UTC](https://revolt.metaeducation.com/forum/t/o-noes-unicode-normalization/1322/9 "2024-06-12T23:26:18Z")

</div>

While reading on some details Unicode normalization, I found a writeup of how URLs could be significantly changed by the process:

[Unicode normalization could change the structure of a URL · Issue #626 · whatwg/url · GitHub](https://github.com/whatwg/url/issues/626)

That's nuts.

To quickly refresh everyone's memory on our URL! datatype: we now think of it as being the decoded form...like what people would copy and paste out of a browser's URL bar. That could contain forms not legal to GET or POST to a web server. So the percent-encoding is something one has to choose to do with a string conversion to or from a URL! if applicable:

[Treat URL! as normal strings with no encoding behavior by hostilefork · Pull Request #655 · metaeducation/ren-c · GitHub](https://github.com/metaeducation/ren-c/pull/655)

However the GitHub issue I link up top points out that normalization is another nuance that is a place for bugs / hacking / etc. This jumped out to me as a very simple--yet insidious--thing:

> NFD and NFKD will normalize the following three characters to generate potentially forbidden code points:
> 
> - `\u2260` (`≠` as one code point) to `=\u0338` (`≠` as two code points)
> - `\u226E` (`≮` as one code point) to `<\u0338` (`≮` as two code points)
> - `\u226F` (`≯` as one code point) to `>\u0338` (`≯` as two code points)

So you aren't supposed to have **\<** or **\>** or **=** in the URL, but they can pop up as part of the unicode normalization process if you decompose things.

**[I've been pushing against automatic transformations as a broken concept--even when it was just carriage returns and line feeds.](https://revolt.metaeducation.com/forum/t/newlina-non-grata/1207)**

Here I have the same instincts creeping in. So what do we know?

- **We should not be building a system that normalizes or denormalize anything without being asked.**

- **The bias of stored formats on disks and filesystems is NFC**

- **When you think about streaming, e.g. bytes flowing one by one from your program to something like the Windows or Linux terminal to display, decomposed forms give rise to chaos**

This is making Latin1 seem like a saner choice, and that engaging Unicode has actually been a mistake.

 ![the_history_of_unicode](https://revolt.metaeducation.com/forum/uploads/default/original/1X/7d1580ef3d9d786d8a847bdaaf9bb606c8251134.png)

## Is "NFC Everywhere" Possible?

We already enforce atomic manipulations to strings at the byte level so that you can only insert/remove byte sequences that leave the string as valid UTF-8.

The canonization process is standardized, and ostensibly we know what the [combining characters](https://en.wikipedia.org/wiki/Combining_character) are. Wouldn't be trivial to implement, but we could say all modifications must generate NFC.

```
>> string: "e"
== "e"

>> append string #{CC81} ; the accent combining character in BINARY!
** Error: Modification to string would create non-canon form

```

That's different from today's answer, which lets you do it:

```
>> append string #{CC81}
== "é"

```

I'm not particularly afraid to go in this direction, but there are some glitches.

I gather not all combining character sequences people might use canonize to a single codepoint. Some meaningful sequences to readers are only available as their decomposed forms, and normalization just ensures that the combining marks are in the same byte order.

This means that if you're in the position of the terminal implementer, then so long as you are dealing with a character that could be composed with something in non-canonized form you'd have to wait for a non-composable character (?) before you print anything... or be able to go back. It seems to me nonsensical that the combining characters come _after_ the character you combine with. Maybe we should buck the trend with our own format that puts them first, at least in streaming cases. 🙂

Anyway, just wanted to write down some more thoughts about this heinous thing.

---

_[View the full topic](https://revolt.metaeducation.com/forum/t/o-noes-unicode-normalization/1322)._
