Chris's iterator framework inspired another Glon experiment.
We've now implemented a general-purpose, closure-based iterator vocabulary in Glon, without introducing a native ITERATOR! datatype, new binding semantics, looping constructs or changes to the frozen S1 substrate.
The vocabulary provides it/bytes, it/peek, it/peek2, it/next, it/pos, it/skip, it/take and explicit end-of-stream detection.
A lexer built on these iterators produces the same token representation as our earlier native Glon Pratt parser.
Results on the 65,536-cell Glon reference machine:
-
77/77 iterator tests passed.
-
28/28 lexer tests passed.
-
1,200 differential token sequences comprising 11,307 tokens matched the original Pratt lexer exactly.
-
Byte advancement requires no managed-heap allocation.
-
Iterator state survives repeated garbage collections.
-
Traversal across 201 fragments works without linear growth in return-stack depth.
There are trade-offs. The iterator lexer is about twice as slow as the original position-based lexer in our unoptimised native benchmark, although repeated peeking is considerably cheaper. WASM-specific iterator parity remains to be tested.
The interesting result is architectural: Glon's existing closures and contexts were sufficient to implement resumable iteration as a loadable vocabulary. We didn't need any new evaluator machinery.
The implementation and tests are available here:
Our next experiment is to use this iterator-based lexer in the native Glon Pratt parser and see whether the abstraction remains worthwhile in a complete parser.
Thanks, Chris, for providing such an interesting challenge! ![]()