In this post we continue where part 1 left off in our language-architectural in-a-nut-shell

Mathematical and Scientific Fidelity

Language safety isn't just about authority. The type system plays another huge role in different failure modes that a safe language should try to attenuate or prevent. In this part of the nut-shell post we are going to look into the Merg-E numeric type-system and how it extends the safety properties of the language.

Numeric Semantic Safety

The numeric type-system in Merg-E can almost be seen as two completely different worlds that interact with each other in strictly defined ways. On one side of the divide we have a wide family exact numbers that lack any notion of dimensions, but have infinite fidelity at the price of a limited numeric range for their bitwidth, and on the other side of the divide there are the physical quantities represented internally by floating point numbers that have limited fidelity but a very wide range given their bitwidth, and in Merg-E explicitly need to be defined with dimensional information. The first family in Merg-E is considered the Web 3.0 side of the numeric typesystem, extremely suitable for cryptography, while the second family is considered to be the data-flow side of the Merg-E numeric type system, steeped in the needs of real-world physical data safety.

Merg-E has no bare floating point numbers because it considers limited-fidelity numbers without dimensional information as unsafe. That is why we had to write:

    inert nodim rquantity16 pi = 3.140635;

It is a real number with explicitly no dimensions stored in a 16 bit floating point number, but the fact that it has no physical units aka dimensions is semantically relevant.

Merg-E tries to represent number theory on both side of the code numeric-typesystem divide:

Family	Main branch	Number theory	symbol	min with	max width
whole	Web 3.0	Whole Numbers	ℕ	1	16384
integer	Web 3.0	Integers	ℤ	2	16384
rational	Web 3.0	Rational Numbers	ℚ	16	32768
rquantity	Physics	Real Number	ℝ	16	256
gaussian	Web 3.0	Gaussian Integer	ℤ[i]	16	32768
grational	Web 3.0	Gaussian Rational	ℚ[i]	32	65536
cquantity	Physics	Complex Number	ℂ	32	512

Within these families, Merg-E distinguishes between runtime types that always have bitwidths equal to some power of two and compile-time virtual types that can have any arbitrary bitwidth within the defined range.

One important safety property of the Merg-E Web 3.0 numeric typesystem is that overflows are completely prevented by result promotion. A concept called integer bitwidth generics (illustrated earlier by the 128 + 1 example) tries to conservatively determine—using virtual types for intermediate results—the safe return type of all operations that might overflow in other languages. The details of this fall outside the scope of this post.

If a Web 3.0 type would grow beyond the maximum bitwidth of its family, demotion occurs instead of further promotion. The result becomes an rquantity256 or cquantity512 carrying a special empty DEMOTE system of units rather than any of the ordinary unit systems. More on this in the next section.

As we saw in Part one, a programmer may deliberately bypass the normal overflow-type-expansion protection with a hazardous typecast. For rationals and Gaussian rationals this hazardous path has a useful fallback: partial on-overflow canonicalization. Full canonicalization of a rational is an explicit, complete reduction to lowest terms. It is potentially expensive and is therefore offered as a separate operator that should be used sparingly:

    inert rational64 myvar1 = 735 ÷ 1575;
    inert rational64 myvar2 = canonical myvar1;

(735 = 3·5·7², 1575 = 3²·5²·7 → gcd 105 → 7÷15.)

Because the operation can be CPU-intensive for large primes and because timing differences may create side-channel leakage, it is not performed automatically on every arithmetic step and it's usage is discouraged in any cryptographical calculation. When a hazardous typecast would otherwise overflow, however, the toolchain may attempt a partial canonicalization. Reduction proceeds only far enough to keep the result inside the original bit-width (or up to a configurable maximum prime bound). The standard bound is 8161. As soon as overflow is prevented, the process stops. This limited repair is the only automatic use of canonicalization and exists solely as a safety net under the hazardous marker.

Physical Quantities

As we saw in the previous section, real numbers and (real based) complex numbers don't just map to floating points, they are a combination of three things:

• A floating point value
• A system of units
• A list of integers denoting together what physical units (if any) the floating point number(s) represent(s).

We aren't going deep into the dimensional analysis in the scope of this post, the important thing to realize is that if one rquantity represents volts and another represents amperes, then you can't accidentally assign the result of the first times the second to an rquantity that represents ohms because the accidentally multiplied instead of divided.

The systems of units matter too.

Merg-E has three systems of units defined, and a wildcard system. The wildcard system and the DEMOTE system are only allowed to have all zero dimensions. And while the wildcards system can be used in operations with all other three systems, the other three systems can't interact in any operation with any other of these three.

There are SI units, PLANCK units, DEMOTE values and nodim values. You can for example multiply a nodim and an SI rquantity, you can not multiply a PLANCK and an SI, a PLANCK and a DEMOTE or an SI and a DEMOTE value. It won't compile just as assigning a wattage result to a resistance won't compile.

inert sivoltage rquantity16 v = 230;
inert sicurrent rquantity16 i = 0.4;
siresistance rquantity16 r = v * i;

The above code won't compile. Note that the modifiers like sivoltage are in fact aliasses that contain info on both the system of units and the actual dimensions. The language spec defines what these are aliasses for. Elaboration on this is out of scope for this post.

Tensors

While Merg-E lacks tuples and lists, it's numeric typesystem goes beyond just the scalars of number theory discussed above, it also has the concept of tensors. These extend to both sides of the numeric type system. We have actually already seen an example of a tensor because all of our rquantity and cquantity values are actually convenience shorthands for tensors in the Merg-E type-system already, for example:

   inert nodim rquantity16 pi = 3.140635;

is horthand for:

   inert quantity::<tensor::<lfreal16,0>,dimensions::<absolute: false>> pi = 3.140625;

or in shorter intermidiate notation:

    inert quantity::<tscalar.real16, units.none> pi = 3.140625;

And the following code:

    inert silength rquantity16 height = 1.825;

is shorthand for:

    inert quantity::< |||
      tensor::<lfreal16,0>, |||
      dimensions::<sys: SI, length: 1, absolute: false> |||
      > height = 1.825;

This is because scalars are rank-0 tensors. We can co beyond this by creating rank-1 and higher rank tensors:

inert quantity::<  |||
  tensor::<lfreal256,2, [3,3], [symmetric::<[0,1]>], noframe>, |||
  dimensions::<sys: SI, mass: 1, length: -1, time: -2, absolute: false> |||
  > sigma = htensor::<¿,0> [0,¿] § [0,1,2],[1,¿] § [¿,0,3];

Explaining the details of tensors and the link with rquantity en cquantity, and elaborating on the above syntax is too much detail for this post, but the important part to realize is that tensors can either be stand alone and be part of the Web 3.0 branch of the numeric typesystem, or be part of a quantity type and by that be part of the physics branch.

Let's focus on the tensor part:

  tensor::<lfreal256,2, [3,3], [symmetric::<[0,1]>], noframe>

We see the tensor type has (up to) five parameters:

• The underlying scalar type, in this case an lfreal256 (a 256 bit low fidelity real aka float256)
• The order of the tensor, in this case 2, a basic matrix
• The shape of the tensor, in this case 3x3
• A list of symmetries or asymmetries the tensor must adhere to.
• The frame where the tensor exists.

As with errors, it is possible to use one of the pre-defined frames like noframe, probframe or universalframe or claim a frame as a resource.

While non-verbose Merg-E uses rquantity and cquantity and bare floats are not allowed in Merg-E, there is actually a non-idiomatic loophole, because bare float-based tensors are actually allowed given that they are defined either in the probabilities frame (probframe) or in a user-consumed frame:

  tensor::<lfreal256,0,[],[],probframe> mybarefloat =  3.1415; 

Usage of tensors like this is considered non-idiomatic. The loophole exists because in use in statistics, bare floating point tensors play an important role. Use of a floating point scalar type with noframe and universalframe is currently undefined toolchain behaviour. The language spec flags it as invalid, and this may get enforced in the future, or this may be a thing that edsger will flag but that muscle will allow. One philisophical stance that should be noted here is that Merg-E doesn't refer to for example a float32 as a float32. It refers to it as a lfreal32, or a 32 bits low fidelity real because floats are considered just that, a low fidelity standin for a real number.

    alias $scope.export.weightframe = consume resource.frames;
    inert tensor::<lfreal256,2,[512,512],[],weightframe> layer_weights = ambient.settings.dothfjson "INNUENDO_SAMPLE_WEIGHTS";

Some random weird language quirks worth mentioning.

literal operators and non-ASCII characters in Merg-E

As you may have noticed in the tensor literal:

 ... =  htensor::<¿,0> [0,¿] § [0,1,2],[1,¿] § [¿,0,3];

Merg-E employs a lexically stratified grammar that extends beyond standard 7-bit ASCII. The core language utilizes the Latin-1 Supplement block (U+0080 to U+00FF) for its primitive operations. For example, the ¿ character (the symmetry operator) and the § character (the sparse section operator) reside within this foundational single-byte stratum. This partitioned character space provides a richer set of single-token operators while maintaining easy keyboard entry via standard compose sequences (e.g., compose ? ? for ¿, and compose s o for §).

Elaboration of these two operators is out of scope here, but there are simpler infix literal constructors that illustrate this design:

  inert gaussian16 myvar1 = 17 ¡ 48;
  inert rational256 myvar2 = 11 ÷ 59;
  inert isolength rquantity128 myvar3 = 1.437 þ nano;

• The first line is a literal for the Gaussian integer 17 + 48i, the ¡ character (compose ! !) is the complex compose operator.
• The second is the literal for the rational number 11/59, the ÷ character (compose : -) is the rational compose operator.
• The third is a scaled literal for 1.437 nano meter, that maps to 0.000000001437. The þ character (compose t h) is the metric scaling operator.

This encoding-bound namespacing extends even further. While the core language's built-in operators are strictly confined to the base single-byte Unicode space, Merg-E provides an extensible lexical architecture for users. The language allows developers to claim unreserved code points from higher Unicode planes (such as specific ranges within the Basic Multilingual Plane) to define custom operators. This mechanism safely extends the grammar without risking collisions with the core namespace.

The exact implementation details are out of scope here, but you could, for example, allocate the ℧ code point as a user-defined operator, making the following expression perfectly valid:

    inert whole32 bar = baz + 7 ℧  17 14 * quaz + 5;

Mandatory Ratliff style indentation

You may have already noticed the indentation style. If you are a developer under 45, you might consider it a bit weird, but this is the Ratliff indentation style. Before C++ and Java took over the world and standardized K&R-style braces for the masses, Ratliff and K&R were on fairly equal footing in the C community. While it is mostly a die-hard choice in certain C circles today, in the opinion of Merg-E's creator, it remains by far the easiest way to keep track of brace pairs. It essentially combines the visual clarity of Python's indentation with the explicit boundaries of C's braces.

Because Merg-E is highly oppinionated and because we think that mixed styles make for hard to maintain code, and Ratliff combines the good parts of C braces with Python indentation, Ratliff in Merg-E is mandatory and not a stylistic choice. Let's step away from Merg-E for a bit and look at indentation in C. The examples are taken from Wikipedia and do illustrate the differences. We use the C examples to avoid showing too much invalid Merg-E code in this article.

while (x == y) {
    foo();
    bar();
    }

This is Ratliff style, in Merg-E this would still be invalid because Merg-E mandates two space indentation:

%while (x == y) {
  foo();
  bar();
  }

The two spaces are because Merg-E is a closure-only language, and we don't want liberal indenting force developers to resort to using modules only for the sake of line length.

So what are some common examples of indentation style that Merg-E rejects. One of them is the popular K&R style:

while (x == y) {
    foo();
    bar();
}

Other ones are BSD style:

while (x == y)
{
    foo();
    bar();
}

and GNU style:

while (x == y)
  {
    foo();
    bar();
  }

There are other styles too, but I think these four are by far the most popular. In Merg-E only a two space variant of Ratliff style constitutes valid Merg-E:

app myApp lang ambient {
  use ambient {
    io.cout as cout;
    io.endl as endl;
    };
  mutable main ()::{
    cout;
    endl;
    }{
      cout "Hello World" endl;
      };
  };

No comments, documentation as reversed markdown

There is no suvh thing as inline comments in Merg-E. Inline comments put undue pressure on the already horizontal pressure od a closure-only language, and the same reasons for choosing Ratliff indentation are important here. We don't want to force developers to jump to modules and merge contract when local closures make the code more readable.

Instead of comments, Merg-E has reverse markdown as documentation facility. An example:

#!/usr/bin/env merg-e
```userdoc
## myApp

This is a silly App to demonstrate some basic Merg-E syntax. 
The App counts how many prime numbers there are between 1 and 100.
```
app myApp lang resource ambient {
  ```devdoc
  ### imports
  Import the ambient authority needed for outputting text to the user. 
  ```
  use ambient {
    io.cout as cout;
    io.cerr as cerr;
    io.endl as endl;
    };

  mutable main ()::{
    cout;
    cerr;
    endl;
    }{
      inert whole16 max_prime = 1000;
      shared mutable whole16 ok_count = 0;
      mutable blocker all_primes;
      ```impldoc
      #### range errors on is_prime

      Please note that is_prime will throw a range_error when given an integer smaller than 1 or larger than max_prime
      ```
      reentrant mutable merge utils.is_prime as is_prime(x whole16)::{
        ok_count;
        }{
          max_prime;
          }@[range_error];
      take 100 {
        all_primes += is_prime(current + 1 nopromote);
        };
      await.all all_primes;
      cout "counted " ok_count " prime numbers" endl;
      }!!{
        range_error : {
          cerr "Something went wrong" endl;
          }};
  };

Note why it is called reverse markdown. The triple backtics (quadruple is also allowed) within the code delimit a markdown documentation block within the code the same way that they would denote a code block within a regular markdown document. We have three levels of documentation, userdoc for the highest level documentation, devdoc for functional dodumentation aimed at developers looking how to use the code, and impldoc that are basically the equivalent of comments meant purely for developers doing maintenance on this bit of code.

Under the Hood: Pipeline & Runtimes

The Three-Step AST Pipeline and the family of runtimes

The above image depicts the three stage pipeline for creating a (vanilla) runnable parse-tree for the first two of the runtimes, fafnir and nidhogg-v, as well as an added 4th stage needed for the planned per-application runtime nidhogg-e. For other runtimes the pipeline will look a little dufferent after the first 3 stages, but the principles will be the same.

What is important to realize is that in both scenarios we are building a runnable parse-tree. Merg-E runtimes don't aim to be either REPLs or full on bytecode VMs, it is a slow moving one-man project, so we need to be realistic as to how far we can push it. Merg-E runtimes run something called a runnable parse-tree. This is a JSON file, to be precise a HF-JSON file or High-Fidelity JSON file, that contains a parse tree of a Níðhöggr runnable execution DAG.

This runnable parse-tree is created in at least three main steps:

• bones : Bones does so called semantic lexing. It does lexing and just enough parsing, superficial capability tracing, and structural analysis to create canonical pre-parse tree.
• muscle : Does name resolution and full least authority lexical scoping. After this phase, the resulting intermediate parse tree is fully canonical, and all of the parse tree will adhere to the language least authority guarantee.
• skin : Maps the intermidate parse-tree to actorcitos and structural iterators that can run on the Níðhǫggr/workers of the target runtime. It's output is a vanilla runnable parse-tree that tatgets any standard non-extended runtime.

For some run-times there is a 4th stage that looks something like this:

• cyborg : This stage takes the vanilla runnable parse-tree and turns it into two things:
  • An Extended Runtime runnable Parse-Tree. This runnable parse-tree contains invocations of operations and potentially data used by these operations that are not available in a standard run-time instead of chunks of vanilla runnable parse-tree chunks that would do the same things. And in addition cyborg outputs Intermediate Representation (IR) code code that can be used in the runtime build-chain. This IR are basically self-contained compiled Merg-E functions that when compiled together with the vanilla runtime, create an Extended Runtime (ER) where the ER runnable parse-tree can run on.

For completeness, let's look at the blue-sky map of all possible runtimes. Some of these runtimes may never exist, but Merg-E as a language was designed especialy with the potential possibility of the whole spectrum of runtimes in mind:

• fafnir : A testing runtime on top of Python meant for language design evaluation. Runs vanilla runnable parse-trees.
• nidhogg-v : A compiled generic runtime. Runs vanilla runnable parse-trees.
• nidhogg-e : A compiled "extended" runtime. One compiled version of nidhogg-e (the e stands for embedded) must be paired with the ER runnable parse-tree from the exact same cyborg run.
• jormungandr : Like nidhogg-e but with extensive use of SYCL.
• nidhogg-b : A version of nidhogg-v for the BEAM virtual machine.
• fucanglong : An imagined single threaded zkVM runtime. Should be nidhogg-v like, functionally, but aimed specifically at zero knowledge proof of code existence.

Yggdrasil & Níðhöggr, Actorcitos & Structural Iterators

At runtime, the runnable parse-tree is turned into work for runtime workers, by a combination of Yggdrasil , Níðhöggr, actorcitos, and structural iterators.

Yggdrasil is the in-memory graph and membrane database containing the runtime state. Níðhöggr is the scheduling engine that turns the runnable execution graph into executable work and feeds that work to the available runners.

The two language-level constructs most important to that process are actorcitos and structural iterators.

An actorcito is a small executable unit. At the language level it looks much like a function, but its calling convention is designed for incremental execution: an actorcito can process a piece of a structure, produce partial or final results, and have its own error path. At runtime, actorcitos are the things that runners actually execute.

A structural iterator describes how those actorcitos are fed work. Despite the name, these are not conventional language-level iterators that simply produce a sequence of values. They describe a traversal over a Merg-E structure — such as a string, DataFrame, or DAG — together with the actorcitos to invoke and, where needed, filtering or ordering rules.

So, in the simplest terms:

the structural iterator describes the work, Yggdrasil is the structor where the work is queued, the actorcito performs the work, Níðhöggr schedules the work, and the runner executes it.

This is also why constructs that look fundamental in most languages do not have to be fundamental in Merg-E. %if and %while, for example, can be expressed as higher-level abstractions over actorcitos and structural iterators. The same underlying machinery can therefore be used for ordinary flow control as well as for more concurrent or structurally complex execution.

Like many things in Merg-E thingas like '%while' and '%if' are thin abstractions and actorcitos & structural iterators are exposed as power-user language features.

The important consequence is that the runtime does not need a separate special-purpose execution mechanism for every language construct. The runnable parse-tree ultimately reduces execution to a relatively small set of runtime primitives that Níðhöggr can schedule onto runners.

Some v0.5 ideas

Version v0.4 of the language specs already defines a few language-level skin-macros, such as %if, %while, and %take. To make Merg-E friendlier to both OO and functional programmers, v0.5 aims to extend this collection with additional convenience macros that map to existing actorcitos and structural iterators, or to established DAG and membrane construction patterns.

Candidates include %map, %filter, %fold, and other common functional constructs for programmers with a functional programming background. For programmers with an OO background, a macro such as %bundle could make the construction of OO-like sub-DAGs more convenient, without introducing a separate object model.

Some of these macros will need to continue an expression directly on the result of a macro invocation, before that result has been assigned to a name. The standard structure-traversal dot cannot serve this purpose, because applying it to a not-yet-expanded macro invocation would trigger premature (attempted) canonicalization.

For this reason, v0.5 will introduce the macro-rvalue operator » ( compose > > ). Here, an rvalue is the unnamed result of a macro invocation, one that has no DAG path yet. When a skin macro is used as an rvalue, the definition of that macro may give » a macro-specific continuation meaning. Macros that do not define such a meaning do not accept ».

The v0.5 extensions to the Merg-E language spec are still under development, so no code examples are included at this stage.

Finally

I hope this rather long post provided the reader with a single post digest of the core of what is in contained in the 25 part language spec blog post series. Merg-E is still in early development and progress is slow, but the language specs are close to where they can be frozen and my focus can shift to completing hf-json, vaultfs, bones, muscle, skin anf fafnir. So if you are reading this and have any feedback: Please comment on this post, either here on HIVE, or discus it with the ocap security crowd on the cap-talk google group. All input is appreciated.