Rebol's recurring problem was that every signal--"splice this", "there's nothing here", "this failed", "here are several results"--had to be encoded as an ordinary value. Ordinary values can be put in blocks, passed through code that doesn't know about your convention, and mistaken for data.
Isotopes add one orthogonal axis to (potentially) every value. An antiform carries the same payload as its ordinary counterpart, but can't be stored in a list. A total, lossless operation called LIFT maps any value into the storable space (antiforms become quasiforms, everything else gains a quote level), and UNLIFT inverts it.
So signals ride on the value, instead of being baked into variant operations...and they can't contaminate data unless you lift them on purpose.
The rest of this post is the long way around to that conclusion.
Where It Started: /ONLY
Historically, Rebol had a problem of how to append blocks "as-is" vs splicing. This was resolved somewhat unsatisfyingly by splicing blocks by default, and using an /ONLY refinement to append as-is:
rebol2>> append [a b c] [d e]
== [a b c d e]
rebol2>> append/only [a b c] [d e]
== [a b c [d e]]
Years of fretting over the bad properties of this led to a somewhat inescapable conclusion:
It's better to carry the intent of whether a value needs to be spliced on that value...as opposed to having subtle variants of core operations that modulate the splicing.
I'd worked up to a point where I was implementing the "mark of intent" by adding a quoting level to suppress splicing. Yet this faced likely accidents when someone had a quoted value in a variable...and really meant to use it somewhere as-is, with the quote--vs. thinking of the quote as a splice-suppression signal which the operation should remove.
(Keep that problem in mind: the mark lived in the same space as ordinary data, so data could be mistaken for the mark. We'll come back to it.)
Then @rgchris made this remark:
If Trying This In Historical Redbol, What Might One Do?
As a rough first cut, let's represent splices with a specially recognizable 2-element wrapper block. We'll signal it's a splice with a series in the first slot--checking for the unique identity of that series. Then put the block itself as the second element:
splice-cue: "!!!splice!!!"
spread: func [block [block!]] [
return reduce [splice-cue block]
]
splice?: func [value] [
if not block? :value [return false]
return same? splice-cue first value
]
Then we can write our new versions of things like APPEND that are specifically aware of this construct.
append*: func [series [series!] value] [
return either splice? :value [
append series second value
][
append/only series :value
]
]
It works more or less in your average Redbol, e.g. in Red:
red>> append* [a b c] spread [d e]
== [a b c d e]
red>> append* [a b c] [d e]
== [a b c [d e]]
red>> append* [a b c] 'd
== [a b c d]
red>> append* [a b c] first ['d]
== [a b c 'd]
In fact, this is essentially how the bootstrap executable for Revolt simulates the SPREAD behavior.
But the weaknesses are immediately apparent!!! ![]()
Not A Distinct Type: Too Easy To Overlook Handling
There's no special type for the spliced block...it's just a BLOCK!. This means any routine that hasn't been written to handle it, will just let it leak through.
red>> reduce [spread [a b c] [a b c]]
== [["!!!splice!!!" [a b c]] [a b c]] ; not [a b c [a b c]]
Changing to some other generic type that can contain a block...such as an OBJECT!...doesn't help matters. You are kind of in trouble any time an operation willfully lets you put these into an array.
The first instinct might be to introduce a new SPLICE! datatype, with a system-wide rule that splices can't be put into arrays. (Enforcing such a rule across all array-manipulating code is challenging...so let's sort of make a note of that fact, but continue.)
Because of the peculiar nature of not being able to be put in a block, there'd have to be a decision made about function arguments as to whether or not they took this type. Many functions designed to handle generic values would not be able to handle them, so there'd presumably need to be some typeset like ANY-NOTSPLICE! or ANY-NORMAL!.
How To Represent A Type That Can't Be Put In A Block?
Now we've got several things to ponder about our new type. For instance: what you should see here?
>> obj: make object! [foo: spread [d e]]
== make object! [
foo: ???
]
We just said that a defining feature of SPLICE! is that you can't accidentally put them in blocks. But the argument to MAKE OBJECT!, namely [foo: ???], is a block. If ??? can't itself be a splice!, then what is it?
This brings up a possibly-related question: what if you want a way to put the intent of whether to splice or not into "suspended animation?"... in a way that you could collect it?
Here's a sort of contrived example of the puzzle:
generate: func [n [integer!]] [
if even? n [return reduce [n n + 1]]
return spread reduce [n n + 1]
]
lift: func [
return: [...]
in [splice! block!]
][
...
]
unlift: func [
return: [splice! block!]
wrapped [...]
][
...
]
n: 0
pending: collect [while [n < 4] [keep lift generate n, n: n + 1]]
data: copy []
for-each item pending [append data unlift item]
How would you write LIFT and UNLIFT such that at the end of the code above, you'd get:
>> data
== [[0 1] 1 2 [2 3] 3 4]
If the system didn't provide some answer to this, you'd end up needing to re-invent something kind of equivalent to the primitive ["!!!splice!!!" [...]] mechanic as a means of persistence:
>> pending
== [[0 1] ["!!!splice!!!" [1 2]] [2 3] ["!!!splice!!!" [3 4]]]
...and then you'd re-invent it again for the next kind of signal, and the one after that.
Isotopes Turned Out To Be The Answer
Isotopes weren't originally invented for splicing. They were introduced to address issues like what an UNSET! was...which has the same class of problems as SPLICE! (not wanting to be put in BLOCK!s, not accepted by default or by most routines).
That they fit splicing without modification is the best evidence I have that the generalization is real, and not something retrofitted to each new problem.
Isotopes introduce two new variants of datatypes called antiforms and quasiforms. Antiforms cannot be put in blocks. Quasiforms can, and evaluate to antiforms.
Isotopes are:
-
general - most base value types (e.g. unquoted things with no sigils) can potentially have antiforms and quasiforms
- (Note: While the -mechanics- are generalized, we restrict the creation of antiforms for states that do not have meaning yet. For example: antiform INTEGER! has no meaning today, but an interesting meaning might come up... so until one does, we don't let you make them. See the Table of Antiforms in Use for the current state of affairs.)
-
efficient - marking a value as an antiform or quasiform never allocates. It's merely a different state of a byte in the value cell (the same byte that encodes quoting levels). Compare the wrapper approach above, which allocated a new block for every SPREAD.
-
"meta-representable" - all antiforms can be produced by evaluating their quasiforms, and quasiforms can be produced by evaluating quoted quasiforms.
"Why Not Just Add A SPLICE! Type?"
That's the right question to ask, and the answer is why this is generalized isotopes and not a pile of special types:
-
One LIFT/UNLIFT works for every signal. LIFT turns antiforms into quasiforms and gives everything else a quote level. That's total (works on every value) and lossless (UNLIFT always gives back exactly what you had). So the "suspended animation" problem above is solved once, for splices, nulls, failures, multi-returns, and whatever antiform shows up next year. With a pile of special types, each one needs its own persistence story.
-
The payload is reused. A splice is a block's payload in a different state. Length, iteration, binding--all the machinery that already works on blocks works on it. There's no parallel SPLICE! implementation to keep in sync.
-
There's one enforcement point. I mentioned at the outset that it would be costly to bulletproof all of native code against putting a SPLICE! into a block. With isotopes, the code that keeps antiforms out of arrays is written once, and covers every antiform--current and future.
-
It's the same byte as quoting. Quote levels, quasi, and anti are one encoding. That's why LIFT is cheap, and why it composes with quoting instead of fighting it.
And The Quoting Problem From The Beginning?
Recall the reason quoting-as-splice-suppression was abandoned: a quote mark lived in the same space as ordinary data, so a quoted value you meant could be mistaken for a signal you didn't.
Antiforms don't have this problem, because they live outside the space of things that can be stored in lists. You can't accidentally hold a splice in a block. You can only hold its lifted form--the quasiform--which is inert until something deliberately evaluates or unlifts it. The signal and the data can no longer be confused, because they no longer share a representation.
Answering The Questions
Above I asked:
What you should see here?
>> obj: make object! [foo: spread [d e]] == make object! [ foo: ??? ]
Isotopes give us the answer, that it's foo: ~[d e]~. That's the QUASIFORM! of BLOCK!, which when evaluated produces an antiform of BLOCK!...which by convention represents a splice.
And because LIFT is lossless, the representation never has to guess. A field holding a splice and a field holding a plain quasiform come out distinct:
>> obj: make object! [foo: spread [d e], bar: '~[d e]~]
== make object! [
foo: ~[d e]~
bar: '~[d e]~
]
But antiforms themselves have no canon representation. The console can print out a comment or show them in a different color, but to talk about them having a representation doesn't make much sense as you'll never see them in source. On the forum I write them between backslashes, which is not LOAD-able syntax:
>> ~[d e]~
== \~[d e]~\ ; antiform (splice!)
I also asked:
"How would you write LIFT and UNLIFT such that at the end of the code above, you'd get:"
>> data == [[0 1] 1 2 [2 3] 3 4]
With block antiforms representing splices, you don't need to write LIFT and UNLIFT... because they come in the box. And instead of using blocks to wrap, they use the super-efficient QUASIFORM! and QUOTED!, so the pending array would look like:
>> pending
== ['[0 1] ~[1 2]~ '[2 3] ~[3 4]~]
When the QUOTED! blocks are UNLIFT-ed, they become regular blocks and then are appended as-is. When the QUASIFORM! blocks are UNLIFT-ed they become antiforms and give the SPLICE! intent. This produces the desired "suspended animation".
And the leak from the Red version?
>> reduce [spread [a b c] [a b c]]
== [a b c [a b c]]
And when an antiform reaches a list without being triaged, you don't get silent contamination. You get an error, and you have to say what you meant:
>> append [flag:] ~null~
** Error
>> append [flag:] boolean ~null~
== [flag: false]
That error is the "one enforcement point" paying off. The same check rejects every antiform, whether or not the code doing the appending has ever heard of it.
Three Classes Of Value
Not all antiforms are equally restricted, and the restrictions form a ranking:
| In lists? | In variables? | Examples | |
|---|---|---|---|
| First-class | yes | yes | plain, QUOTED!, QUASIFORM! |
| Second-class (stable antiforms) | no | yes | NULL, OKAY, SPLICE!, ACTION! |
| Third-class (unstable antiforms) | no | no (they decay or escalate) | PACK!, FAILURE!, TRASH! |
The line between second and third class buys an important guarantee: anything you can store in a variable can be conditionally tested. IF doesn't need a ^META parameter to be sure its condition won't blow up.
The Proof Is In The Capabilities
I've explained about splices (antiform BLOCK!), and mentioned how it crosses needs with unset variable states.
-
NULL is implemented as an antiform of WORD!. It's the one falsey state, and since it can't be put in blocks, you'll never trip over one in your data. That makes enumeration trustworthy: the only null array access can hand you is the one from running off the end.
>> block: [a b false c] >> while [value: try take block] [print mold value] a b false cIn Rebol2 and Red, "is there an element here?" and "is the element NONE!?" are the same check, so the classic
any [pick block index, <fallback>]idiom has never been sound. -
FAILURE! (the antiform of ERROR!) is a sneaky out-of-band way to return definitional errors.
-
ACTION! (the antiform of FRAME!) is what we'd traditionally think of as an action/function. It triggers execution if accessed via WORD! references. This makes it safe to handle items picked out of blocks without worrying about defusing actions...because only quasiform or plain frames can be put in blocks in the first place!
-
PACK! (the antiform of GROUP!) implements multi-return values. A plain assignment decays it to its first value, so callers who don't know about the extra results never see them. (If that sounds exotic, Common Lisp's multiple values have decayed to the primary value in single-value contexts since the 1980s. What's new is that a pack has a lifted form you can store, inspect, and hand back.)
What It Costs
It's more vocabulary than Rebol ever asked of anyone. Generic routines have to decide whether they accept antiforms, which is what ^META parameters are for. And the names have churned a lot getting here--see the history.
But the distinctions themselves aren't invented. "Splice this or not", "nothing here vs. a value", "the loop broke vs. the body returned none" all existed in Rebol2 and Red. They just had nowhere to live except inside the programmer's head.
It's natural for there to be some confusion with the new idea--especially given all its churn through the course of design. But the design is becoming clearer, and I think people are going to find this gives solidity to writing complicated but coherent code...vastly outpacing historical Redbol.