This is part twentyone in a series on the language spec for the Merg-E Domain Specific Language for the InnuenDo Web 3.0 stack. Where most previous posts were about v0.3, this is the second post that is about the v0.4 version of the spec. I'll add more parts to the below list as the spec progresses:
- part 1 : coding style, files, merging, scoping, name resolution and synchronisation
- part 2 : reverse markdown for documentation
- part 3 : Actors and pools.
- part 4 : Semantic locks, blockers, continuation points and hazardous blockers
- part 5 : Semantic lexing, DAGs, prune / ent and alias.
- part 6 : DAGs and DataFrames as only data structures, and inline lambdas for pure compute.
- part 7 : Freezing
- part 8 : Attenuation, decomposition, and membranes
- part 9 : Sensitive data in immutables and future vault support.
- part 10 : Scalars and High Fidelity JSON
- part 11 : Operators, expressions and precedence.
- part 12 : Robust integers and integer bitwidth generic programming
- part 13 : The Merg-E ownership model, capture rules, and the --trustmebro compiler flag.
- part 14 : Actorcitos and structural iterators
- part 15 : Explicit actorcitos, non-inline structural iterators, runtimes, and abstract scheduler pipeline.
- part 16 : async functions and resources and the full use of InnuenDo VaultFS
- part 17: RAM-Points, RAM-points normalization bag, and the quota-membrane.
- part 18: Literal operators & Rational and Complex numbers.
- part 19: Interaction between operators, integer bitwidth generics, and the full numeric type-system.
- part 20 (v0.4): Compile-time dimensional analysis, SI/Planck units and the scaling literal operator.
- part 21 (v0.4): Tensors and tensor literals.
- part 22 (v0.4): Deprecating float/complex for rquantity/cquantity for full dimensional type-safety.
- part 23 : Flow control and exceptions.
- part24: The merge parts of Merg-E
- part25: The $threshold keyword and logging.
In this post, we are going to look deeper into tensors and htensors. In Merg-E, tensors and htensors (higher order tensors) are closely related but fundamentally distinct. Tensors exist purely at compile time as a rich typing mechanism, resolving to highly optimized runtime scalars, vectors, and n-dimensional htensors.Tensors can exist stand-alone, or embedded as part of a quantity (as discussed in our previous post). Stand-alone higher order tensors can derive from any numeric type and are meant primarily for use with our five exact type families: whole, integer, gaussian, rational, and grational, predominantly within a cryptography context. (Please note that the v0.4 language spec renames the uint type family to whole, turning the old uint tokens into aliases).In contrast, tensors embedded within a quantity are designed specifically for physics-related simulations and calculations. Consequently, they are restricted to deriving exclusively from our two inexact type families: lfreal and lfcomplex.
We discussed quantity in depth in the previous post, so in this post we are going to take a deep dive into the tensor and just the tensor. But we'll use quantity in some of the code examples.
Why tensors and why the exact/inexact split?
Merg-E is a Web 3.0 dataflow and crypto DSL. Dataflow often uses physics fields and math. For that reason Merg-E needs the concept if a quantity with lfreal or lfreal based lfcomplex as underlying atomic numbers, and with dimensional analysis for physics alligned type safety as discussed in the previous post. Next to that, crypto often uses matrices and lattices tha fit well on tensors and the type safety Merg-E lets tensors providen, hence stand alone tensors that use exact types like integer and rational as underlying atomic numeric type. The two are distinct but they overlap.
Tensors vs htensors
Once a tensor passes compile-time checks, including dimensional analysis, symmetry validation, and frame compatibility, it lowers to either a plain numeric scalar or a htensor. At runtime, there are no unit tags, no frame labels, no symmetry metadata. The safety is enforced entirely at compile time, leaving zero runtime cost.
the general syntax.
Let's go back to our example from the previous post, the stress tensor:
mutable quantity::< |||
tensor::<lfreal256, 2, [3,3]>, |||
dimensions::<sys: SI, mass: 1, length: -1, time: -2, absolute: false> |||
> sigma;
We discussed the unit bit extensively, so let's zoom in to the tensor section and let's expand it a little bit to capture the full type annotation container:
<tensor::<lfreal256, 2, [3,3], [symmetric::<[0,1]>], noframe>
Note that the tensor annotation has four parts:
- The atomic type the tensor builds on
- The tensor rank, 0 for a scalar, 1 for a vector, or in this case 2, for a 2-dimensional matrix, etc.
- The shape of the tensor, in this case a 3x3, so we have a 3 by 3 matrix.
- A symmetries specification.
- An optional coordinates frame specification
symmetry specifications
The symmetry specification needs a bit of extra attention. It is a collection of zero or more axes of (a)symmetry, in our rank-2 tensor there is only one possible collection of two indices offer what a symmetry or asymmetry can exist, the 0,1 collection, but a rank-3 or rank-4 can have multiple axes of (a)symmetry. In this case, the stress tensor, a symmetric entry, tells us the stress tensor needs to be symmetric over the 0,1 axis, and asymmetric tensors over that axis can't be assigned to it. Other tensors may define asymmetric instead. In that case symmetric tensors on that axis can't be assigned to it. If neither symmetric or asymmetric is defined, both are permitted and no compile errors will occur on that axis.
In addition to standard symmetry, we also recognize skew symmetric ssymmetric, hermitian semmetric hsymmetric and skew hermitian symetric shsymmetric. Please note that htensor literals are currently underspecified for hermitian semmetric and skew hermitian symetric tensors. They are currently mentioned for competeness but an aditional post on hsymmetric and shsymmetric tensor literals will need to resolve this.
frame specification
We define an optional tensor type modifier to specify a frame. This is meant as an other type safety field for tensors that helps avoid mixing tensors that shouldn't mix. Unfortunately because Merg-E does not have user defined modifiers, we will revert to aliasses to simulate them. This unfortunately leaves open the potential for subtle bugs, but is still considered better than not allowing for frame specification.
Merg-E 0,4 defines the following frames:
- noframe: Just a tensor, no coordinate system
- universalframe: A considered to be universal coordinate system
- localframe: A considered to be local coordinate system
- userframe128 .. userframe255: 128 distinct but genericly named coordinate systems
It is considered non-idiomatic to directly use the userframeNNN designations. Instead the creation and usage of aliasses is considered idiomatic use.
alias $scope.export.inertialframe = $scope.export.lang.tensor.frame.userframe128;
alias $scope.export.bodyframe = $scope.export.lang.tensor.frame.userframe129;
Then if you make sure not to alias other frames to the exact same userframe03 and userframe04, you can use inertialframe and bodyframe and be sure they will never be used together without leading to compile errors.
It is likely more language defined frames will follow, especialy those relevant for crypto or physics, but for now only noframe, universalframe and localframe are defined. We currently have 125 frame slots (003 .. 127) available for extending the language defined frames and limit the need for aliases in the future.
typesets as a resource
In v0.4 we define the $scope.resource sub tree (for now) containing the following three nodes:
- rampoints
- frames
- errors
We leave rampoints and errors for now and consider frames for now. In the previous section we used an alias to $scope.export.lang.tensor.frame.userframe128, which gave us the risk of collisions. Using the resource interface we can prevent any collisions by invoking consume on a resource. This works for frames and errors (not for rampoints).
alias $scope.export.inertialframe = consume resource.frames;
alias $scope.export.bodyframe = consume resource.frames;
This does the exact same as the earlier alias commands, but using this API, collisions are prevented.
Because like rampoints, frames and errors are a resource now, we can also use a quata proxy as with rampoints:
mutable dag newdag = membrane<quota<resource.frames, 16, 2, 4>> olddag;
What the above statement does is that it defines newdag as a quota-membrane over olddag with a total quota of 16 frames user types. The second number claims 2 user frames for direct allocations with the membrane itself, and the third number defines that layered membranes may be created with a total overcommit of 4 user frames. This means the maximum quota pool available to hand out to directly stacked child membranes is calculated as: (Total Quota - Direct Allocation) + Overcommit.
htensor literals
As we discussed a few times before, we reserved the latin-1 unicode space for literal operators. These are operators we need to write special literals in Merg-E. Htensor literals are a little bit more flexible than other literals. They allow us all kinds of sparse htensor definitions. A htensor literal is a comma separated list of htensor chunks, so let's start of with looking what a simple sparse htensor literal looks like:
[¿,¿] § [[0,1,2],[1,0,3],[2,3,0]]
We can write the same like this:
[0,¿] § [0,1,2],[1,¿] § [¿,0,3],[2,¿] § [¿,¿,0]
Or like this:
htensor::<¿,0> [0,¿] § [0,1,2],[1,¿] § [¿,0,3]
Or a bit more readable with the line concatenation pipes:
htensor::<¿,0> [0,¿] § [0,1,2], |||
[1,¿] § [¿,0,3]
So let's investigate what is going on.
First let's look at the section operator § (type using compose s o) . The section operator works on two section lists, the section scope list and the section definition list. Basically it reads like LH scope has RH definition. In the first example the whole htensor is one single scope. In the second and third example each row has its own sub scope.
The second literal operator is the inverted question mark operator ¿ (type using compose ? ?). This operator has a different meaning in the scope list than it has in the definition list and htensor modifier that we will look at next. In scope context the ¿ is a placeholder for every single value of a given index, so [¿,¿] denotes the scope of the entire rank-2 tensor while [0,¿] denotes the scope of just row 0- of that tensor.
This should explain the first form:
[¿,¿] § [[0,1,2],[1,0,3],[2,3,0]]
This is the compact form, we define the htensor literal as one big non-sparse compact chunk.
On the definition list side, as well as in a htensor modifier, the ¿ operator is the symmetry operator. It is a placeholder that should use symmetry to fill in the actual value. Note that the use of this operator here is only possible if the tensor has a single axis of symmetry. Let's look again at the second form:
[0,¿] § [0,1,2],[1,¿] § [¿,0,3],[2,¿] § [¿,¿,0]
Note there are three chunks, one for each row now. Note that the ¿ operators in the definition list keep us from having to duplicate values that can make use of symmetry.
Now for the third form:
htensor::<¿,0> [0,¿] § [0,1,2],[1,¿] § [¿,0,3]
In this for the literal is prefixed with a htensor modifier that modifies the entire expression. The htensor modifier has two attributes:
- The default value for non axis-of-symmetry values
- The default value for axis-of-symmetry values.
The use of this modifier in this case allows us to expand on the sparseness of the tensor. We don't need to define row 2 because the defaults give us enough info to infer it. The third row is not needed here because the whole [2,¿] § [¿,¿,0] chunk from the second example can be filled in from the two defaults.
Bringing it together
Let's bring this back to our original example line, and change from mutable to immutable:
inert quantity::< |||
tensor::<lfreal256,2, [3,3], [symmetric::<[0,1]>]>, |||
dimensions::<sys: SI, mass: 1, length: -1, time: -2, absolute: false> |||
> sigma = htensor::<¿,0> [0,¿] § [0,1,2],[1,¿] § [¿,0,3];
It's all a bit verbose, still, but the type safety this all brings should be worth the verbosity.
Stand alone tensors.
inert tensor::<int8, 2, [6,6], [symmetric::<[0,1]>]> identity6 = htensor<0,1>;
Where quantities are float or complex based, stand-alone tensors are meant for precise types, like int in this example. In Merg-E these are mostly meant for cryptography. This example shows an int8 based tensor that form the 6x6 identity matrix.
Tensors vs matrices
The core distinction between tensors and htensors in Merg-E is that tensors are strictly compile-time constructs; they have no runtime footprint. The compiler uses tensors to enforce strict type-safety, perform dimensional analysis, validate symmetry, and parse rich sparse literals. Once these checks pass, the tensor completely dissolves, folding its data into standard runtime scalars, flat vectors, or multi-dimensional htensors.
A little latin-1 help
As the number of latin-1 codespace characters in Merg-E grows from three to five with this post, a quick cheat sheet as to how to type these four characters on a standard keyboard without layer support.
| Merg-E literal operator | character | Linux | Mac | Windows |
|---|---|---|---|---|
| symmetry | ¿ | compose ? ? | Option + Shift + ? | Alt + 0191 |
| sparse section | § | compose s o | Option + 6 | Alt + 0167 |
| complex compose | ¡ | compose ! ! | Option + 1 | Alt + 0161 |
| rational compose | ÷ | compose : - | Option + / | Alt + 0247 |
| metric scaling | þ | compose t h | Option + T | Alt + 0222 |
If your keyboard supports layers, you are strongly advised to take advantage of your hardware's custom mapping capabilities. For example, I use a split ergonomic keyboard (a ZSA Voyager), which allows me to group these four Merg-E literal operators together right on my home row for maximum efficiency.
Layer 0 is my default layer.
Layer 1 is mostly my function keys layer plus some often used characters, I've grouped the four latin-1 Merg-E literal operator characters together with the square and curly brackets on this layer.
Layer2 is my standard navigation layer.
And finally I'm using layer 3 as my terminator layer, with keyboard shortcuts and macros for using the windowed terminal emulator Terminator on Linux.
This is just my personal keyboard layout, it is just to give an idea of how to conveniently configure a layered keyboard for development and how to make it as Merg-E friendly as possible.
conclusion
In this post we discussed an important v0.4 language spec part in this series, tensors and tensor literals. We looked into basic symmetries as typesystem invariants that increase type safety, and we discussed sparse vs dense literals.
I'll keep working on the v0.4 spec, but my current priority lies with the implementation of the testing runtime, so don't expect many posts in this series in the coming months.
