# What To Do About Horrible, Grievous, Unicode

**URL:** <https://revolt.metaeducation.com/forum/t/what-to-do-about-horrible-grievous-unicode/2280>\
**Category:** Philosophy\
**Created:** [September 15, 2024, 1:38pm UTC](https://revolt.metaeducation.com/forum/t/what-to-do-about-horrible-grievous-unicode/2280 "2024-09-15T13:38:04Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![hostilefork](https://revolt.metaeducation.com/forum/user_avatar/revolt.metaeducation.com/hostilefork/32/421_2.png) [@hostilefork](https://revolt.metaeducation.com/forum/u/hostilefork)\
**Post date:** [September 15, 2024, 1:38pm UTC](https://revolt.metaeducation.com/forum/t/what-to-do-about-horrible-grievous-unicode/2280/1 "2024-09-15T13:38:04Z")

</div>

At some point [I picked up a test from Red](https://github.com/red/red/blob/master/tests/source/units/series-test.red#L1413C2-L1417C39), which is basically this:

```
str: " ^(A0) ^-a b ^- c ^(2000) "
assert ["a b ^- c" = trim copy str]

```

That's some funny business.

- The hex character ^(a0) is decimal 160, which is the standard Unicode translation for `&nbsp;`... non-breaking space.

- The hex character ^(2000) is "EN QUAD".. [a space that is one _en_ wide: half the width of an em quad.](https://en.wikipedia.org/wiki/Quad_(typography))

R3-Alpha (and Revolt) never had support for trimming these characters out. So the test fails.

But it wound up flying under the radar, somehow.

_(I think I didn't actually pick up the file it was in to the tests until some time after adding it. And by the time I added it, a lot of things were breaking due to binding and I was putting off fixing everything until it was time. Now it's that time--I'm going item by item reviewing breakages and getting the tests in order.)_

But when I got to this one, the log just said:

```
(   
    str: " ^-a b ^- c "
    "a b ^- c" = trim copy str
) "failed, test returned null"

```

Because I didn't go look up the test (I thought I had it right there)...I didn't realize there was funny business because the display doesn't give you any indication. Neither does Red's after the transcode:

```
red>> " ^(A0) ^-a b ^- c ^(2000) "
== " ^-a b ^- c "

```

Even pasting it into VS Code (which I didn't, until just now) gives you _terribly weak_ feedback that something weird is going on:

![image](https://revolt.metaeducation.com/forum/uploads/default/original/1X/fcb694d9fa08aa3be6b625b1529de82cf9e2ecef.png)

Gee. Glad I had "show invisibles" turned on--that _really_ did a lot for me there. 🙄

_(Seriously, what is the point of that feature if that's what it's going to do?)_

## I Don't Want This Stuff In Source Files

We can't fix the world. They're doing what they are doing. This stuff is the currency of text and you have to support it.

But we can set house rules. The default mode for Revolt should only allow two invisible characters in source: space and newline. (And I'd like there to not be space at the end of lines.) This would be a hard rule for any script in official repositories.

I'd have saved myself an hour of confused digging if there'd been an error when I pasted in the console, telling me I was dealing with a messed-up situation. There'd have to be some conscious shift into a mode to tolerate it... temporarily as some kind of way to import a string into the system.

## Not Ready To Support This Test

There's a [sparse bitset implementation](https://revolt.metaeducation.com/forum/t/roaring-bitset-s/1327) that has been on the shelf, but that's needed before we create unicode charsets for high codepoints.

Anyway, there are higher priorities. But I definitely do feel like there should be some alarms going off when you are reading files with disruptive codepoints. You should have to say "Yes, I want ugly codepoints" or "Yes, I want emoji".

A totally permissive TO TEXT! operator shouldn't be what people are reaching for. You should have to be explicit. **`(decode [@utf-8, whitespace: all, emoji: all] blob)`**. Principle of least privilege... conservative defaults.

The names for the specializations should help guide behavior. **`(decode @utf8-unchecked blob)`**. (Unnatural?)

- "What does that mean, unchecked (looks up documentation)"
- "Oh, I have an option to have it screen out weird whitespace? Wow! Great! "

Something like **`utf8-basic`** would make conservative choices--the same ones used by default for source code.

---

<div class="post-metadata">

**Author:** ![bradrn](https://revolt.metaeducation.com/forum/user_avatar/revolt.metaeducation.com/bradrn/32/365_2.png) [@bradrn](https://revolt.metaeducation.com/forum/u/bradrn)\
**Post date:** [September 16, 2024, 12:34am UTC](https://revolt.metaeducation.com/forum/t/what-to-do-about-horrible-grievous-unicode/2280/2 "2024-09-16T00:34:43Z")

</div>

> [@hostilefork](#):
>
> The default mode for Revolt should only allow two invisible characters in source: space and newline.

I think this is a sensible choice. Whitespace other than the usual ASCII space just causes confusion in source code.

> [@hostilefork](#):
>
> But I definitely do feel like there should be some alarms going off when you are reading files with disruptive codepoints. You should have to say "Yes, I want ugly codepoints" or "Yes, I want emoji".

This, on the other hand, worries me. Both non-breaking spaces and en quads have legitimate uses in text — they’re hardly ‘ugly’ at all. (There’s even more obscure ones with legitimate uses: e.g. text in French uses the narrow non-breaking space, U+202F.)

In general, it feels to me that you’ve been trying to manage the complexities of Unicode by ignoring them. (First by trying to normalise everything, now by trying to remove annoying characters entirely.) The trouble is, Unicode is complex because the problem domain of text is complex. Speaking as someone who spends a considerable amount of time dealing with multilingual text, I can say with confidence that trying to avoid the issues does nothing but make them more complex when you inevitably stumble upon them.

As for this test itself: some parts of Unicode are complex, but not this. The [Unicode character database](https://www.unicode.org/Public/16.0.0/ucd/UnicodeData.txt) lists precisely 17 characters with general category `Zs` (`Space_Separator`), plus one line separator and one paragraph separator. That’s 19 characters in total. I see no reason why Revolt couldn’t recognise them.

---

<div class="post-metadata">

**Author:** ![hostilefork](https://revolt.metaeducation.com/forum/user_avatar/revolt.metaeducation.com/hostilefork/32/421_2.png) [@hostilefork](https://revolt.metaeducation.com/forum/u/hostilefork)\
**Post date:** [September 16, 2024, 11:19am UTC](https://revolt.metaeducation.com/forum/t/what-to-do-about-horrible-grievous-unicode/2280/3 "2024-09-16T11:19:09Z")

</div>

> [@bradrn](#):
>
> I think this is a sensible choice. Whitespace other than the usual ASCII space just causes confusion in source code.

Then you agree there are domains in which it makes sense to be conservative.

The question is: how often are usages of usermode string values based on domains where conservatism makes sense?

Maybe I'm unusual--in that I work on an interpreter, so most "usermode" strings I work with wind up being source code at some point. If I worked on other kinds of programs that had to pass-through arbitrary freeform HTML or emoji-laden SMS messages all the time, I might have a different bias.

I have strong feelings about [Freedom To vs. Freedom From](http://blog.hostilefork.com/freedom-to-and-freedom-from/). e.g. I'm the sort of person who is completely convinced that allowing spaces in filenames caused far more harm than good to the field of software.

Anyway, for right now my principal agenda is to stop losing hours on this 💩 by not letting it bite me every time it comes up.

I don't know what the exact tradeoffs to be made here are, or which you'd agree with. But I give the example of pasting things into the console. And that can be a situation like you type **`string: -{`** `<<PASTE>>` **`}-`** and I don't want to get arbitrary invisibles when you do that without some warning. Perhaps you'd think that falls under the "it's okay to be restrictive there, because it's source code".

Anyway, it doesn't necessarily need to be _hard_ to put the console into a mode where it allows such pastes ( **`console @utf8-unchecked`** )...I just don't want it to be the default. And I want there to be an established vocabulary for talking about these domains.

Once that vocabulary exists, I want people to use it. I've mentioned that I'm tired of all the implicitness in the system, it just breaks every time a change comes along. How the 32-bit to 64-bit change was handled in R3-Alpha and Red is a perfect example of how people love to just kick the can down the road with more bad choices after having just seen the prior methodology fail.

* * *

> _A COBOL programmer, tired of all the extra work and chaos caused by the impending Y2K bug, decides to have himself cryogenically frozen for a year so he can skip all of it._
> 
> _He gets himself frozen, and eventually is woken up when several scientists open his cryo-pod._
> 
> _"Did I sleep through Y2K? Is it the year 2000?", he asks._
> 
> _The scientists nervously look at each other. Finally, one of them says "Actually, it's the year 9999. We hear you know COBOL."_

---

<div class="post-metadata">

**Author:** ![bradrn](https://revolt.metaeducation.com/forum/user_avatar/revolt.metaeducation.com/bradrn/32/365_2.png) [@bradrn](https://revolt.metaeducation.com/forum/u/bradrn)\
**Post date:** [September 16, 2024, 3:19pm UTC](https://revolt.metaeducation.com/forum/t/what-to-do-about-horrible-grievous-unicode/2280/4 "2024-09-16T15:19:27Z")

</div>

> [@hostilefork](#):
>
> Maybe I'm unusual--in that I work on an interpreter, so most "usermode" strings I work with wind up being source code at some point. If I worked on other kinds of programs that had to pass-through arbitrary freeform HTML or emoji-laden SMS messages all the time, I might have a different bias.

Yes, I think this is the unusual domain showing. In my kinds of programs, by contrast, I very regularly have to deal with multilingual text and ‘unusual’ (to English-speakers) Unicode characters. I’d say my situation is probably more representative of the more common use-cases.

What helps me in writing these programs? Basically, a language which doesn’t get in my way when I manipulate these strings, but only does exactly what I tell it to. Arguably Haskell takes this to an unusual extreme: I can’t even directly read a file to a string, I need to read it to a byte array and then parse that as UTF-8. But it does make my job immensely easier — I can port my program to Windows. Linux and WebAssembly without having to worry that locales or OS conventions will mess everything up. That’s a win in my book.

> [@hostilefork](#):
>
> And that can be a situation like you type **`string: -{`** `<<PASTE>>` **`}-`** and I don't want to get arbitrary invisibles when you do that without some warning. Perhaps you'd think that falls under the "it's okay to be restrictive there, because it's source code".

I lean towards allowing arbitrary Unicode in strings — especially ones with such distinctive delimiters. The benefits of being able to copy-paste arbitrary text outweigh the possibility of that text having an unexpected character. (Remember, this test case was _designed_ to be unusual… most text isn’t like this.)

Instead, on reflection, I tend to see this case as a problem with _printing to the console_. When Haskell prints strings for debugging purposes, it escapes all non-ASCII characters by default:

```plaintext
ghci> "unicöðe texŧ"
"unic\246\240e tex\359"

```

Why can’t Revolt do the same? It would nicely solve the problem of copying from the console, while leaving the language itself unaltered.

---

<div class="post-metadata">

**Author:** ![hostilefork](https://revolt.metaeducation.com/forum/user_avatar/revolt.metaeducation.com/hostilefork/32/421_2.png) [@hostilefork](https://revolt.metaeducation.com/forum/u/hostilefork)\
**Post date:** [March 18, 2025, 11:27pm UTC](https://revolt.metaeducation.com/forum/t/what-to-do-about-horrible-grievous-unicode/2280/5 "2025-03-18T23:27:27Z")

</div>

> [@bradrn](#):
>
> > [@hostilefork](#):
> >
> > Maybe I'm unusual--in that I work on an interpreter, so most "usermode" strings I work with wind up being source code at some point. If I worked on other kinds of programs that had to pass-through arbitrary freeform HTML or emoji-laden SMS messages all the time, I might have a different bias.
> 
> Yes, I think this is the unusual domain showing. In my kinds of programs, by contrast, I very regularly have to deal with multilingual text and ‘unusual’ (to English-speakers) Unicode characters. I’d say my situation is probably more representative of the more common use-cases.

At least on the _encoding_ side of UTF-8, I have an answer for stopping the propagation of weird bytes if you want, that's now pretty palatable...

I was fretting over how TO of BINARY! doing UTF-8 encoding wasn't really fitting nicely with my idea of it, and so I preferred to go ahead with a more explicit parameterization to ENCODE:

```
>> encode 'utf8 -{C😺T}-
== #{43F09F98BA54}

```

It's not that much longer than **to binary!** , just two characters:

```
to binary!
encode 'utf8

```

But you get a lot more information. However, if you have these personal settings, it gets a little bit messier, let's say like:

```
>> encode [utf8 emoji: no] -{C😺T}-
** Error: Emoji disabled in UTF-8 Codec configuration

```

That's a lot to type so you're liable to need to invent a name for it, which puts some burden on you. But **utf8-of** isn't a terrible name for you to call your specialization:

```
/utf8-of: specialize encode?/ [codec: [utf8 emoji: no]]

```

This would override the default configuration, we'll assume UTF8-OF is in LIB is defined as some sane default.

It's still a little bit ugly due to the hyphen, BUT, pursuant to the newfound **[binding-based configurability of infix OF](https://revolt.metaeducation.com/forum/t/squaring-the-circle-of-length-and-length-of/385/8)**, you can call the relevant local definition for your environment with no space! And still not lose your variable names!

```
>> utf8: utf8 of -{C😺T}-
== #{43F09F98BA54}

>> utf8
== #{43F09F98BA54}

```

And maybe the default UTF8-OF offers some refinements to swap up the settings, that OF can migrate onto the call:

```
>> utf8: utf8:basic of -{C😺T}-
** Error: Codepoints of ENCODE UTF-8 with Basic setting Must Be < 65535

```

So things like UTF8:PRINTABLE could be options of the stock UTF8-OF, and you could just say in your file:

```
/utf8-of: utf8-of:printable/

```

Then all your **utf8 of** would be protected, in the scope of that file (or if you imported your standard configuration library, in all the files that use your personal configuration... I expect everyone to have such files... the goal here is to _bend the language to you_, not to bend yourself to the language...)

🪄

This strategy, of having strong routines like ENCODE or CHECKSUM, which can be parameterized and specialized, ultimately being able to "hook" things like OF, is looking like it's going to be able to hit the target of the kind of flow we are seeking.

So if you want the exact bytes, you can always use AS BINARY!, which just aliases the memory as a binary blob directly. Anything else is making a copy, and while memcpy() would be fastest a little bit of filtering isn't going to be a problem.

## What About the Decoding Side?

That... I dunno yet.

So there's **`decode 'utf8`** With the corresponding options like **`decode [utf8 emoji: no]`** or whatever. More thinking is needed on how READ and LOAD and other such operations get configured.

But at least I think the encoding side looks good.

---

<div class="post-metadata">

**Author:** ![hostilefork](https://revolt.metaeducation.com/forum/user_avatar/revolt.metaeducation.com/hostilefork/32/421_2.png) [@hostilefork](https://revolt.metaeducation.com/forum/u/hostilefork)\
**Post date:** [March 16, 2026, 12:29pm UTC](https://revolt.metaeducation.com/forum/t/what-to-do-about-horrible-grievous-unicode/2280/6 "2026-03-16T12:29:31Z")

</div>

> [@hostilefork](#):
>
> ## I Don't Want This Stuff In Source Files
> 
> We can't fix the world. They're doing what they are doing. This stuff is the currency of text and you have to support it.
> 
> But we can set house rules. The default mode for Revolt should only allow two invisible characters in source: space and newline. (And I'd like there to not be space at the end of lines.) This would be a hard rule for any script in official repositories.

Not that it requires any deep thought to have known this, but... **invisible unicode characters are--indeed--quite dangerous** :

> **[Glassworm Returns: Invisible Unicode Malware Found in 150+ GitHub Repositories](https://www.aikido.dev/blog/glassworm-returns-unicode-attack-github-npm-vscode)**
>
> The Glassworm supply chain attack is back. Researchers uncovered malware hidden in invisible Unicode characters across 150+ GitHub repositories, plus npm packages and VS Code extensions.
