scrobble.life
HiveDevs

A language-architectural in-a-nut-shell of the Merg-E language. Part 1/2

Rob J Meijer@pibara (68)37d

While I have posted a long series of language spec posts on my ongoing efforts on the Merg-E language, runtime(s) and tool chain, a good in-a-nutshell overview of the language is still missing. I hope this post can serve that purpose. Even if I may be a bit too close to effectively detach myself from the details, this is my best attempt. It may be more of a language-architectural in-a-nut-shell, but that is probably good as development is still in an early stage and community feedback right now is more important than showing people how to code in a language without a complete toolchain or runtime.

Due to the per post character limit, I need to post this post in two parts even if I would have prevered to do it in just one.

Merg-E: Why another language? Why a DSL?

Why a new programming language in the first place? Aren't there already too many of those? Well to understand the why, it is first important to understand that Merg-E doesn't try to be a general purpose programming language. It is trying to fill a specific place as a Domain Specific Language or DSL. Or rather it is designed to be a DSL bridging two very much joinable domains: Web 3.0 blockchain technology and high throughput non-low-latency data processing. And more specifically, it is trying to do this within the context of the InnuenDo Web 3.0 technology stack. A stack that is primarily being developed around Python and C++ running underneath Python. But a stack in dire need of a security first language that aligns better with the stakes involved in Web-3.0 based data processing than the likes of languages like Python and TypeScript.

And Merg-E isn't just the language. It is a (set of) runtime(s) as well, and that is probably the bigger reason for designing a new security-first language from the ground up. In Python, an asynchronous Python framework called aiow3 is being ported from the old aiohivebot project, and it is a bot library for Web 3.0 chains that maximally connects to API nodes of distributed blockchains. Rather than choosing to dabble with and maintain the messy language bindings for languages like Elixir and Monte, and port aiow3 to both these languages and maybe more, Merg-E attempts to take a minimal subset of what it considers the best parts of Monte and Elixir and their runtimes, and build a new minimalist security first data friendly language where the aiow3 functionality plus other web 3 and data related core functionalities are part of either the language or the runtime.

Merg-E above all other language quirks, is an object capability language, even if the language has no actual notion of objects. It has closures instead. It inherits much of its language philosophy yet almost none of its syntax nor semantics from the E language. But safety in Merg-E goes beyond just least authority, it extends into ownership safety, extreme type safety and semantic safety as well. Most of this in a way that security experts might frown at, because Merg-E provides some "fix later" hooks and "informed deviation" possibilities that are part of the Merg-E philosophy that while it strongly promotes that the safest way should always be the least verbose way, extreme security shouldn't get in the way of prototyping, and sometimes a "fix later" that stands out will allow developers to keep velocity and still only push to production when these fix-later points have eventually been addressed.

While the first Merg-E runtime won't live up to it, the language is also designed for the type of data intensive workflows you may expect from languages like Elixir and like some data flow platforms. But that won't come out until alternate runtimes get developed.

Radical minimalism

A three keyword language; canonical vs non-canonical Merg-E

The canonical form of the Merg-E language has only three keywords. Keywords in Merg-E are recognizable by the $ sign they are prefixed with. And the interesting part is that in most of your Merg-E code you won't see many usages of these keywords. That is because almost everything in Merg-E is a Directional Acyclic Graph (DAG) and two of the three keywords are in fact the root of such a DAG. Merg-E code comes in two forms:

  1. Canonical Merg-E code
  2. Non-canonical Merg-E code

Normally you will be using non-canonical Merg-E code and it will feel like a language with many keywords, but in reality it is not. Non canonical Merg-E is a language ruled by how Merg-E name resolution maps to the DAG that lives under the $scope keyword.

Let's look at some canonical Merg-E boilerplate:

$scope.app myApp $scope.import.lang $scope.import.ambient {
  $scope.import.lang.use $scope.import.ambient {
    io.cout as cout;
    io.endl as endl;
    };
  $scope.import.lang.modifier.mutable $scope.import.lang.main ()::{
    $scope.export.cout;
    $scope.export.endl;
    }{
      $scope.export.cout "Hello World" $scope.export.endl;
      };
  };

The above is the canonical code for a simple hello-world in Merg-E. You will normally not ever write canonical Merg-E code, but you might see it while debugging because the runnable parse-trees that Merg-E uses are reversible to canonical Merg-E not to non-canonical form.

The non-canonical form looks a lot more readable:

app myApp lang ambient {
  use ambient {
    io.cout as cout;
    io.endl as endl;
    };
  mutable main ()::{
    cout;
    endl;
    }{
      cout "Hello World" endl;
      };
  };

Even in its non-canonical form, a few distinct Merg-E design choices stand out: mutable main explicitly marks the application entry point as holding state, while ::{ cout; endl; } defines the explicit closure captures required to print output. Anything not captured in that block simply cannot be accessed inside the function body.

You will note the non-canonical form lacks a number of prefixes, in this case:

  • $scope
  • $scope.import
  • $scope.import.lang
  • $scope.import.lang.modifier
  • $scope.export

The reason why these prefixes can be omitted is because of a second Merg-E keyword, and the existence of a sane default for that keyword. The keyword $resolve_order. While using $resolve_order in your own code should be done with great care, it can be a powerful and important tool when trimming the language itself for least authority purposes. But also for example when the language gets updated and name resolution breaks your old program. We can change the resolve order if we need to by using an expression with the $resolve_order keyword:

$resolve_order $scope.export $scope.import $scope.import.lang.modifier $scope.import.lang $scope;

The third Merg-E keyword is the $threshold keyword. That keyword again like the $scope keyword is the root of a DAG. I't purpose in build-dependent conditional code. In its simplest form we can use it for logging and log levels.

app myApp lang ambient {
  use ambient {
    io.log as log;
    io.endl as endl;
    };
  mutable main ()::{
    log;
    endl;
    }{
      $threshold.log.notice log "Hello World" endl;
      $threshold.log.debug log "Hello World was brought to you by Merg-E in an nutshell" endl;
      };
  };

In our toolchain the flag --thresholds controls if these conditional lines get included in the processed code.

merg-e-bones --thresholds log.notice myapp/myapp.mrg build/myapp-bones.json

The above toolchain command would exclude the verbose debug logging from our code.

No objects; closures with explicit capture

As you might have already noticed if you tried to understand the hello world examples above, Merg-E is a closure-based language. What you probably didn't guess is that in terms of programming paradigms, Merg-E is actually a closure-only language. Merg-E has no objects, no global or global-ish functions, everything with executable code has its data access and other code access managed by closure scope, and implicit and explicit closure captures. It all starts in a way with the application scope:

app myApp lang ambient {
  ...
  };

This is a bit different from the strict closure syntax in the rest of the application (or module) code, but what this basically does is that it defines the application myApp and pulls the full lang and ambient tree into the application scope under $scope.import.lang and $scope.import.ambient.

The lang tree being both inert and immutable is implicitly mapped into $scope.import.lang when a first child scope is defined. At the same time ambient is only mapped to $scope.export if it explicitly captures into an inner scope what generally goes against the least authority principle that Merg-E attempts to encourage. In order to allow only part of the ambient tree to be passed on down the chain, we use the use expression:

  use ambient {
    io.cout as cout;
    io.endl as endl;
    };

What this does is that it takes $scope.import.ambient.io.cout and makes it available under $scope.import.cout, and it takes $scope.import.ambient.io.endl making it available under $scope.import.endl.

Now comes the main scope:

  mutable main ()::{
    cout;
    endl;
    }{
      cout "Hello World" endl;
      };

Here we see most of the closure syntax of Merg-E. Main is a little different from regular callables in Merg-E in that there should be exactly one main in an application and main doesn't have the zero assumption parallelism. The main is guaranteed to be synchronous itself even if its children are not, and as such, main is the only place where the limited number of synchronous language operations may be called and synchronous resources in resource and ambient can be accessed.

But let's look at the syntax. The main function is marked as mutable. What this means for a function is that it is a manager function with access to a resource or mutable data or any ambient authority. The empty (), tell us that main has no function arguments. The following ::{} part is the explicit capture block. Where inert non-mutable entries of $scope.export are implicitly pulled into the closure, mutables, resources, sensitive data and ambient authority carying elements need to be explicitly captured into the inner scope.

Let's expand to the full closure syntax:

  mutable main ()::{
    cout;
    cerr;
    endl;
    }{
      mutable function helloplanet (planet string)::{
        cout;
        cerr;
        endl;
        }{
          cout "Hello " planet endl;
          }!!{
            exception.io_error : {
              cerr "Something went wrong" endl; 
              };
            };
      mutable blocker all_planets;
      all_planets += helloplanet("Mercury");
      all_planets += helloplanet("Venus");
      all_planets += helloplanet("Earth");
      all_planets += helloplanet("Mars");
      all_planets += helloplanet("Jupiter");
      all_planets += helloplanet("Saturn");
      all_planets += helloplanet("Uranus");
      all_planets += helloplanet("Neptune");
      await.all all_planets;
      };

In this code we define the inner scope with the function helloplanet that takes a string as function argument. Note that because we don't mark the function as reentrant and it is the only function that gets to manage the ambient authority of the standard IO, we don't need to use locks, but as we will see later, there are many situations where we will. The most important thing we see is that the !!{} expression defines an error map with error handlers for in this case one error: the io_error.

In the rest of main we see a little bit of the zero assumption parallelism of the Merg-E language. More on this in the next section.

No-assumptions parallelism

In the previous section we showed some Merg-E that is valid and safe, but we didn't explain why it was valid and safe. There is an important property of the Merg-E language that flows from the fact that Merg-E is designed to be a multi-runtime language. The first (still incomplete) runtime that will be available will be single threaded running on top of Python tasks and as such will have no real parallelism. Future runtimes will expand on that, not just with worker threads, but also with parts of the code ending up in a SYCL toolchain, or alternatively running on a highly parallel VM such as the BEAM VM.

For now we can forget about that and just focus on two aspects of the code from the last section. Let's look at the code again:

mutable main ()::{
  cout;
  cerr;
  endl;
  }{
    mutable function helloplanet (planet string)::{
      cout;
      cerr;
      endl;
      }{
        cout "Hello " planet endl;
        }!!{
          exception.io_error : {
            cerr "Something went wrong" endl; 
            };
          };
    mutable blocker all_planets;
    all_planets += helloplanet("Mercury");
    all_planets += helloplanet("Venus");
    all_planets += helloplanet("Earth");
    all_planets += helloplanet("Mars");
    all_planets += helloplanet("Jupiter");
    all_planets += helloplanet("Saturn");
    all_planets += helloplanet("Uranus");
    all_planets += helloplanet("Neptune");
    await.all all_planets;
    };

Note that we define the all_planets blocker as a mutable blocker. This blocker is a collection of handles to runnables, and our invocation of helloplanet returns just that, a runnable handle that can only be discarded (fire and forget) or be added to a mutable blocker for synchronization purposes. When helloplanet is invoked, the developer should have zero assumptions about how parallel the runtime is able to run it. The await.all makes sure all runnables are completed before main ends, but on some runtimes it might already have.

We can easily break the above code by making a promise that we don't keep:

mutable main ()::{
  cout;
  cerr;
  endl;
  }{
    reentrant mutable function helloplanet (planet string)::{
      cout;
      cerr;
      endl;
      }{
        cout "Hello " planet endl;
        }!!{
          exception.io_error : {
            cerr "Something went wrong" endl; 
            };
          };
    mutable blocker all_planets;
    all_planets += helloplanet("Mercury");
    all_planets += helloplanet("Venus");
    all_planets += helloplanet("Earth");
    all_planets += helloplanet("Mars");
    all_planets += helloplanet("Jupiter");
    all_planets += helloplanet("Saturn");
    all_planets += helloplanet("Uranus");
    all_planets += helloplanet("Neptune");
    await.all all_planets;
    };

Here we added the modifier reentrant to the function definition of helloplanet. What this does is that it allows the runtime to schedule the different runnables linked to the helloplanet closure on different runners, and if these runners use threads, cout and cerr might collide in their usage, depending on if they are defined as reentrant in the runtime, and endl might hit an ownership issue depending on if it is defined as shared or borrowed in the runtime.

We can fix the above code with semantic locks:

mutable main ()::{
  cout;
  cerr;
  endl;
  }{
    reentrant mutable function helloplanet (planet string)::{
      cout;
      cerr;
      endl;
      }{
        lock(cout, endl){
          cout "Hello " planet endl;
          };
        }!!{
          exception.io_error : {
            lock(cerr, endl){
              cerr "Something went wrong" endl;
              }; 
            };
          };
    mutable blocker all_planets;
    all_planets += helloplanet("Mercury");
    all_planets += helloplanet("Venus");
    all_planets += helloplanet("Earth");
    all_planets += helloplanet("Mars");
    all_planets += helloplanet("Jupiter");
    all_planets += helloplanet("Saturn");
    all_planets += helloplanet("Uranus");
    all_planets += helloplanet("Neptune");
    await.all all_planets;
    };

Now we have safe Merg-E code that will make it through the pipeline, make it through edsger (the pedantic linter) and will run without potential ownership runtime errors. But we need to realize how much freedom we have given the runtime by writing the code like this. The planets may be printed in order, or they may be printed completely out of order. By making the closure function reentrant, we allow the runtime to schedule the runnables however it sees fit, and we should make zero assumptions about things like order between when the different invocations actually get run, or when their locks get resolved compared to those of previous or later invocations of the same callable.

Please note the difference in terminology. In Merg-E, in the above code, helloplanet is a callable, and helloplanet("Mercury") is a runnable. They are related but still very different concepts.

No structs, no lists, no dicts; DAGs and DataFrames as the Only Data Structures

When looking at the canonical form of Merg-E we already exposed that at it's core Merg-E is about DAGs. The $scope is an important DAG and $scope.import and $scope.export are where most program variables and constants live. When we wrote:

  mutable blocker all_planets;

What we did was we created a new node $scope.export.all_planets in our $scope DAG of the type mutable blocker. Most programming languages have data structures like structs and objects, and maps. Merg-E has none of these. Instead Merg-E has DAGs, and in Merg-E a DAG needs to adhere to rules that aren't much unlike those in a file-system, but a bit stricter.

The base transactional DAG in Merg-E is a tree-graph and this hard tree-graph is augmented with aliases that may not point to nodes that would require parent traversal when reached through the main hard tree-graph structure. The tree graph is said to be the hard part of the graph while the aliases are considered to be the soft part of the DAG. It is important that these distinctions are always valid in a transactional sense. While pruning and grafting (called enting in Merg-E, but graft is available as, you guessed it, an alias in the $scope.import.lang part of the $scope DAG), these rules may temporarily be suspended, but at transaction points like closure definition, these rules will either get enforced again, or a runtime error will occur if these invariants can not be restored. In restoring the invariants of the Merg-E DAG structure, soft DAG links may get promoted to hard DAG links or the other way around.

It is possible to emulate an object or a struct or a map with a sub-DAG.

Let's demonstrate. Imagine we have two values:

  inert whole16 max_prime = 1000;
  mutable whole16 ok_count = 0;

If we define these like this, they will both live under $scope.export directly. Let's see what we need to do to put these in a tiny sub-DAG instead. First we create a new non-leaf DAG node:

  mutable dag mydag;

Now we can prune these values from $scope.import and graft them under $scope.import.mydag:

  ent mydag prune export.max_prime as max_prime;
  ent mydag prune export.ok_count as ok_count;

Now $scope.export.max_prime and $scope.export.ok_count are gone and we have a struct-like sub-DAG under $scope.export.mydag that we can pass as a single value to functions that expect a DAG argument. But right now this DAG acts more like a map than like a struct. Let's fix that:

  noleaf freeze mydag;

Now mydag has a frozen structure, but non constant leaf nodes like ok_count can still be updated. So basically if we dont noleaf freeze the DAG we have something like a map, and when we do we get something more like a struct. Now if we create a closure with our mydag, we come close to a single method object. We could go further by making sure all our struct-like DAG members use shared ownership, but this is considered non-idiomatic in Merg-E, so don't do that unless you already absolutely can't avoid using shared mutables.

Maps, structs and objects aren't the only data structures that Merg-E is missing that you will be used to having from other languages. Merg-E also has no tuples, lists or sets. Because Merg-E aims to facilitate data-stream heavy applications, and a big part of the language is built around processing structured data, the dataframe as DAG-node type, and processing of large dataframes are an important part of the language. To keep the language small and minimal, given that this type of data collection was already needed, Merg-E chooses not to duplicate what a dataframe can do and thus doesn't provide native sets, lists or tuples.

Let's see how we construct a simple dataframe. We need to first create an empty dataframe with some structure:

  mutable dataframe myDf int max_prime int counter int ok_count;

And next we need to fill it and freeze it:

  myDf(100 0 0);
  myDf(100 16 42);
  myDf(800 0 1);
  freeze myDf;

In Merg-E dataframes are mutable by appending only during construction. So no mutable lists or sets, at least not directly. But we can define processing functions for a dataframe that can produce a new immutable dataframe as a result. So transformation rather than mutation.

Thin abstractions, the price of minimalism that has it's merits

When in Merg-E some language feature requires an abstraction between a powerful implementation and a language level abstraction, we explicitly choose to make the abstraction as thin as possible and expose the underlying implementation to language power-users. This currently includes operators, that Merg-E allows to be extended with custom user defined operators in different unicode regions for custom operators, the exposure of scheduling primitives called actorcitos and structural operators that are abstracted by common flow control abstractions like %while and %if but are exposed to power users so they can gain more fine grained error handling that the standard Merg-E abstractions don't deliver. And things like integer generics. All these subjects are too advanced to go deep into in an in-a-nutshell context, but their existence is important to mention because it discloses an important language design philosophy that you will see in more subtle ways throughout the language.

In Merg-E minimalism in implementation and abstraction thickness makes the language harder to use, especially for new users, and it makes the language actually feel less minimal than it actually is because the guts of the implementation are available. This also has huge merits for power users, and hopefully if you read this and it triggers your curiosity, you will one day become a power user.

Extreme Safety by Default (POLA)

We have already seen the Principle Of Least Authority in action with the decomposition and explicit capture of parts of the $scope.import.ambient sub-dag, and we have seen a little bit of freezing, but there is more, much more. In a rather long section we are going to look at the most important POLA language features.

Explicit captures

We already saw that ambient authority requires explicit capture through the closure's explicit capture block, but how about standard scalars, functions, dataframes, etc? Well. Merg-E tries to follow the principle of least authority there. If something is either sensitive or mutable, capture needs to be explicit. If something is inert, capture is implicit. So how does this translate to code:

  whole8 maxcnt = 42;
  string prefix = "http://";
  mutable whole32 counter = 0;
  function myfunc ()::{
    }{
      ...
      };
  myfunc();

Here we have a constant 8 bit whole number named maxcnt, a 7 character string named prefix and a 32 bit whole number named counter. The myfunc function has no explicit captures, so which ones of the three values would you think would be available? To better know how to know this, we need to look at a little quirk of Merg-E immutable defaults.

In Merg-E an immutable is either inert or sensitive, and the default when not specified depends on the capacity of the type to carry something large enough to constitute a secret. For Merg-E basically anything of any of these types is implicitly considered inert:

  • bit
  • whole2
  • whole4
  • whole8
  • quadstate
  • integer4
  • integer8
  • char
  • byte

So looking at this we can read the above as:

  inert whole8 maxcnt = 42;
  sensitive string prefix = "http://";
  mutable whole32 counter = 0;
  function myfunc ()::{
    }{
      ...
      };
  myfunc();

This means only maxcnt will be available to the inner scope of the closure. It is probably obvious by the value that prefix doesn't need to be sensitive, so it should be OK to write this as:

  inert whole8 maxcnt = 42;
  inert string prefix = "http://";
  mutable whole32 counter = 0;
  function myfunc ()::{
    }{
      ...
      };
  myfunc();

The "Move-by-Default" Ownership Model

Now what if in the above example we need counter too? We can just make the function mutable add counter to the explicit capture body:

  inert whole8 maxcnt = 42;
  inert string prefix = "http://";
  mutable whole32 counter = 0;
  mutable function myfunc ()::{
    counter;
    }{
      ...
      };
  myfunc();

But it is important to realize what this means. In Merg-E move is default when explicitly capturing in a closure or when invoking a function with mutable arguments. So after the first myfunc invocation, counter is going to be an empty shell in the outer scope and will only ever be available in the inner scope.

If move is a bit too drastic, the extreme on the other side is declaring counter as shared

  inert whole8 maxcnt = 42;
  inert string prefix = "http://";
  shared mutable whole32 counter = 0;
  mutable function myfunc ()::{
    counter;
    }{
      ...
      lock(counter) {
        ...
        };
      };
  myfunc();
  lock(counter){
    ...
    };

Now counter is available both in the inner and outer scope, but we should probably use a semantic lock before accessing it at least if we plan to mutate it.

Between these extremes we can also go for a third option by declaring counter as borrowed:

  inert whole8 maxcnt = 42;
  inert string prefix = "http://";
  mutable blocker counter_blk;
  borrowed mutable whole32 counter = 0;
  mutable function myfunc ()::{
    counter;
    }{
      ...
      };
  counter_blk += myfunc();
  counter_blk += myfunc();
  counter_blk += myfunc();
  await.all counter_blk;
  ...

Note that now instead of locks we need an await. A borrowed mutable will temporarily move to the inner scope whenever there are active or pending runnables that claim the mutable, then snap back to the outer scope once all runnables have been awaited. This means basically that the outer scope needs to wait for the inner scope to finish before it can safely use the borrowed mutable again itself.

The 'merge' Contract

A Merg-E application will consist of a single mrg application file and zero or more mrm module files. Because Merg-E is a closure-only language, and because it is a least authority language, there is no top level; import like in many languages. Instead Merg-E comes with a surgical least authority import facility defined by a two-sided merge/def contract.

Let's move the helloplanet function into its own hello module. First lets look how the function looked as a normal inner scope function:

  mutable function helloplanet (planet string)::{
    cout;
    cerr;
    endl;
    }{
      cout "Hello " planet endl;
      }!!{
        exception.io_error : {
          cerr "Something went wrong" endl; 
          };
        };

Now let's convert it to a def in a module file:

ns hello lang {
  mutable def helloplanet (planet string)::{
    cout: callable;
    cerr: callable;
    endl: modifier;
    }{
      }@[]{
        cout "Hello " planet endl;
        }!!{
          exception.io_error : {
            cerr "Something went wrong" endl; 
            };
          };
  };

A few things should stand out. First that we need to define what the type is of the different captures in the explicit capture body. Secondly that we have an extra (empty) capture body and the weird looking @[] part. Because we are dealing with multiple files, we can't just use implicit captures, all captures that are actually used by the def will need to be explicitly captured and the second capture block is there to make some implicitly captured entities part of the contract. The @[] bit defines what errors the code might throw and is also part of the contract.

Now in the application file we replace the function definition with a merge definition:

mutable main ()::{
  cout;
  cerr;
  endl;
  }{
    mutable merge hello.helloplanet as helloplanet(planet string)::{
      cout;
      cerr;
      endl;
      }{}@[];
    mutable blocker all_planets;
    all_planets += helloplanet("Mercury");
    all_planets += helloplanet("Venus");
    all_planets += helloplanet("Earth");
    all_planets += helloplanet("Mars");
    all_planets += helloplanet("Jupiter");
    all_planets += helloplanet("Saturn");
    all_planets += helloplanet("Uranus");
    all_planets += helloplanet("Neptune");
    await.all all_planets;
    };

There needs to be a full match between the def and the merge fingerprint for the contract to be valid. This surgical import allows for the closure-only language to still be modular.

Note that modules can not have access to resources or ambient authority through any other means than the contract. The module code has file inherited access to lang, but not to either ambient or resource as application file code can have. As such, all ambient authority is injected into the module code through captures and function arguments.

Sensitive Data & VaultFS Symbiosis

We already touched on the subject of sensitive vs inert constant data. When writing Merg-E code you will quickly realize the price of secure being the least verbose option. By default all literals assigned to a type big enough to hold a secret the size of basically a 4 digit pin code is considered sensitive unless explicitly marked as inert. 

  inert whole16 max_prime = 1000;
  inert string webprefix = "http://"

But how about sensitive data, say a password:

  string password = "SsstDontTellAnyone1948@";

Or explicitly:

  sensitive string password = "SsstDontTellAnyone1948@";

It may come as a shock that this won't compile. By default Merg-E considers in-code sensitive literals as insecure coding practice. We can make it compilable by adding a hazardous modifier, but even then it won't immediately be accepted by the toolchain without additional command line arguments, we will look into this deeper when discussing the hazardous modifier:

  hazardous string password = "SsstDontTellAnyone1948@";

Administered sensitive code-data in VaultFS

On the Linux platform, the primary platform that Merg-E targets to run on, Merg-E has a symbiotic relationship with the Innuendo VaultFS userspace file-system collection. InnuenDo vaultFS allows putting sensitive data into a vault that in theory is only accessible to a single Merg-E program (if this is actually true depends on the specific runtime).

If the vault is used in a read-only way, the simplest way to access vault-stored values is using a vault expression.

  string password = ambient.settings.vault.innuendofs "innuendo-sample-password";

On systems where VaultFS isn't available, a similar expression can be used in a less secure way to extract a sensitive value from the environment:

  string password = ambient.settings.env "INNUENDO_SAMPLE_PASSWORD";

Or from an application settings file:

  string password = ambient.settings.dotconf "INNUENDO_SAMPLE_PASSWORD";

Enough options to avoid the use of hazardous in your code.

With VaultFS, the standard vault expression is meant only for static data in the vault that isn't supposed to change during the runtime of the application. A somewhat more complicated way to get the sensitive data from the vault is using the async resource pattern:

  function  myFunction (user str password str)::{
    }{
      ...
      }!!{
        resource_error : {
          .. 
          }};
  blocker mf;
  mf += myFunction("innuendo", async ambient.async.vault "innuendo-sample-password");
  await.all mf;

What the above pattern does is that myFunc is scheduled conditionally delayed on the resolving of the async call to ambient.async.vault. Once ambient.async.vault extracts the "innuendo-sample-password" sensitive value, myFunc will get invoked with that value.

Administered sensitive runtime-data in VaultFS

So far we only looked at sensitive data that gets exposed to our actual own code. InnuenDo VaultFS also supports an even safer way to handle certain forms of sensitive data. If all the code does with a sensitive data value is invoke runtime functionality, then exposing it to the actual code isn't least authority. Instead VaultFS uses distinct zones for code-data versus runtime-data. The below code is from a yet unstable API but illustrates the concept:

  callable::<string, callable::<string>> signer = ambient.signer.ecdsa "hive-posting-key@pibara";
  done = blocker;
  done += signer(myTransaction, hivePoster);
  await.all done;

In this code we give the string "hive-posting-key@pibara" to a callable factory function ambient.signer.ecdsa. That factory function is part of the runtime and as such has access to the runtime zones of VaultFS. It extracts an ECDSA signing key for the user @pibara for signing HIVE transactions with ECDSA. The signing key is never exposed to the code, only to the runtime.

Next to the administered zones of VaultFS, VaultFS also has self-custody zones. Discussion of these falls outside of the nutshell context of this already quite extensive post.

Pruning, Enting, membranes and caretakers

Now we come to the section that may feel familiar to the Object-Capabilities crowd, but much of the familiar ocap terminology is superimposed on the Merg-E DAG-based nature. We already saw examples of pruning and enting in the struct-emulation section:

  inert whole16 max_prime = 1000;
  mutable whole16 ok_count = 0;
  mutable dag mydag;
  ent mydag prune export.max_prime as max_prime;
  ent mydag prune export.ok_count as ok_count;
  noleaf freeze mydag;

You can do much more with enting and pruning than just do things directly in the $scope.export space. One particularly interesting place to prune and ent in is $scope.import.lang . You can prune and ent your own little subset of the Merg-E language with some effort and some extra aliases. A full discussion of this falls outside of the scope of this in-a-nuthell post. And another interesting use of enting and pruning is the creation of ambient bundles. Again here examples would distract more from the in-a-nutshell slice of the language that this mention, but the base idea is that a full application can have quite a wide range of ambient authority elements it will want to pass down explicitly and passing it as a much smaller set of constructed DAG bundles can make the code much more readable. Care should be taken though to adhere to POLA while constructing ambient bundle DAGs.

One thing to realize when enting and pruning and adding or removing aliases is that in many cases a sub-DAG isn't fully a bare sub-DAG. Instead a sub-DAG will often be a membrane or a stack of membranes on top of the primordial DAG. It is important to realize that these membranes are usually closer to something like an in-memory version control repository like git, than a typical ocap membrane. So when you are enting and pruning away on your version of $scope.import.lang, you are not topworking the global lang sub-DAG but on some possibly copy-on-mutate DAG on top of that primordial DAG.

Other than membranes provided automagically by the runtime, there are membranes the user can directly create that come closer to the membranes from ocap theory.

Freezing

We already glimpsed at freezing in the above code example. Freezing is a tricky little devil when it comes to least authority. Freezing is an important tool for attenuation and least authority. You have something mutable, you initialize it and freeze it, and it's effectively a non mutable (be it with ambiguous sensitive/inert status). But the power to freeze itself is quite a powerful authority that you likely want to attenuate.

For this, Merg-E has a wide range of freezable least authority tools. First, for a mutable to be freezable, the freezable modifier allows us to define freezable mutables. Without it a mutable isn't freezable:

  freezable shared mutable int ok_count = 0

But defining a shared or borrowed mutable doesn't mean every inner scope needs to have the freezable version of the shared or borrowed mutable. So by default Merg-E will insert an nonfreezable membrane, or in Merg-E parameterized primitive notation a membrane::. You could try to use such a membrane yourself manually:

  myblocker += my_function( membrane::<nonfreezable> mydag );

But this is actually automatic. Merg-E considers inserting nonfreezable membranes for freezables as standard least authority, so the above is equivalent to:

  myblocker += my_function( mydag );

Instead if we want more authority including the authority to freeze, we need to write:

  myblocker += my_function( no_proxy::<nonfreezable> mydag );

Other membranes

Exploring all available Merg-E membranes falls out of the scope of this post, but available membranes are:

  • nonwritable
  • nonprunable
  • nonentable
  • nonlinkable
  • nontopworkable
  • nonmutable
  • copyonmutate
  • versionable

And finally there is the quota membrane, but we come back to that one later.

Caretakers

Another name from ocap theory that this time is a bit closer to what is expected from theory than the membrane.

Let's say we have a function called myfunction. We can wrap this function in a caretaker by calling the Merg-E quotacaretaker function:

  caretaker revokeMyFunction = quotacaretaker myfunction;

The caretaker is a special type of callable that when called rewires itself to no longer give proxy access but perform a null operation instead. You need to fetch the actual proxy from the caretaker by calling the Merg-E revocable function on the caretaker:

  proxy::<string> revokableFunction = revocable revokeMyFunction;

That function returns a callable with the same argument types as those of the wrapped function. Now revokableFunction can be called many times until revokeMyFunction gets called and scheduled.

The function quotacaretaker is called that because next to a normal caretaker it can produce one that auto-revokes after a quota amount of invocations. To do this, the function takes an extra non zero whole number argument:

  proxy::<string> quotadFunction  = revocable quotacaretaker myfunction 8;

Now after 8 invocations of quotadFunction the quotadFunction will automatically be revoked.

RAM-points and types as resources

Not everything in Merg-E is a closure or a DAG. Next to the lang and the ambient DAG that we can pull into our application from the application file scope, there is a third DAG available: the resource DAG:

app myApp lang resource ambient {
  ...
  };

This DAG offers different things that represent language level exaustable resources that are too language or runtime bound to be part of ambient and are not immutable enough to be part of lang.

If something is considered a resource, we can use it inside of a quota membrane. The quota membrane is a bit more involved because they are parameterized at two levels and because they need you to get a good mental model of the Merg-E resource over-commit model of thinking.

  mutable dag newdag = membrane::<quota::<resource.rampoints, 80, 40, 30>> export; 
  function somefunction (...)::{...}{...}!!{...} newdag;

The above example shows a quota membrane for a concept called RAM-points. A full elaboration on RAM-points falls outside of the scope of this post, but in short RAM-points are a concept that tries to address the different per type storage needs across runtimes in a way where a typical Merg-E function would roughly use the same ballpark amount of RAM-points across runtimes even if the total amount RAM used might differ greatly between runtimes. So a function that uses 100 RAM points on runtime A is likely to use maybe 75 .. 125 RAM-points on a different runtime, even if 100 RAM points on system A is 400 kB and on the other runtime 2MB or 80 kB.

So what do the three numbers in a quota membrane definition actually mean?

Well, because quota's are applied to resources, and resources are usually under utilized, Merg-E quota's allow for multi layer over commits.

What the quota statement does is that it defines newdag as a quota-membrane over $scope.export wit a total quota of 80 RAM points. The second number claims 40 RAM-Points for direct allocations with the membrane itself, and the third number defines that layered membranes may be created with a total overcommit of 30 RAM points. This means the maximum quota pool available to hand out to directly stacked child membranes is calculated as: (Total Quota - Direct Allocation) + Overcommit.

So the following will be valid:

  mutable dag newdag1 = membrane::<quota::<resource.rampoints, 80, 40, 30>> olddag; 
  mutable dag newdag2 = membrane::<quota::<resource.rampoints,60,30, 10>> newdag1;
  mutable dag newdag3 = membrane::<quota::<resource.rampoints,40,40, 0>> newdag2;

RAM-points aren't the only resource that can be used with quota membranes. An important second type of resource is entropy as a resource. We skip that one in this in-a-nutshell because it goes a bit too deep into cryptographic primitives, but the idea is that a single high entropy key can be used with an index to derive reasonably secure derived keys, and the index space of this function can be treated as a resource. One third resource we will go into deeply is the type as a resource concept.

Two good examples of types as resource are errors and tensor frames. We will discuss tensor frames when we get to the Merg-E type system later in this post, so let's look at error types now. Merg-E doesn't come with user defined types, what is a problem if you need to something akin to defining something like a custom exception type in other languages. What Merg-E does provide is an error type as a resource. Merg-E has a number of built-in error types that have a meaningless name, but that can be aliassed under strict conditions. The user can't alias them directly, but she can alias them in a so called consume action that is held accountable to the currently active scope quota:

  alias $scope.export.invalid_dimension_error = consume resource.errors;

Just like rampoints, errors and frames can be used in quota membranes.

After reading this section, the observant reader may have noticed the absence of something like scheduling points or processing points. The current Merg-E specification does not yet define scheduling points. Scheduling points are being considered as a possible future addition to the resource model, but they are not part of the current language specification. It is not sure yet if a runtime-agnostic measure like RAM-points would make sense, or if any sense of scheduling points will by definition require runtime specificity. As such introduction of scheduling points is delayed until different runtimes have actually been implemented.

The "Hazardous" Hatch & --trustmebro

We already ran into the hazardous modifier when looking at sensitive data literals. Now we will see that hazardous is a multi purpose code red-flag that conveys either a fix-later in prototypes or programmer intent when conceptually less pure language constructs are being used. We will skip hazardous in user defined operators because it is an advanced topic, but the below are the prime uses of hazardous you are bound to use or encounter.

hazardous prune

When we discussed pruning and enting we skimmed over aliasses, but aliasses become relevant when we look into pruning. We need to ask:

  • "what happens to an alias when we prune a part of the DAG that has incoming edges from aliases?"

Well, that depends. A standard hard prune leaves aliases dangling in the parent scope, and will trim any dangling aliases in case of subgraph inheritance by deeper execution contexts. This means the edges are considered to be lexical in nature, giving the user the chance to ent an alternative target for the edge before hand over to deeper execution scopes.

Alternatively the user can opt for a soft prune, defensively named a hazardous prune:

  hazardous prune $scope.import.lang.Amplify;

To understand the above we need to realize that $scope.import.lang.ent and $scope.import.lang.graft are both aliasses that point to $scope.import.lang.Amplify.ent.

When we mark a prune operation as hazardous we ask prune to aggressively restructure the DAG in such a way that other than the removed edge all graph invariants are preserved IF IT CAN, and to prune a minimum amount of stale aliases if it can't. There are no guarantees that all runtimes or even subsequent versions of the same Merg-E runtime will make the same decisions in this or will have the same restructuring intelligence, hence the marking as hazardous of soft prunes.

In this example the semantic lexer will have two options. Either the lang.ent alias will get deleted and be replaced by lang.Amplify.ent and the lang.graft alias will get updated to point to lang.ent, OR the lang.graft alias will get deleted and be replaced by lang.Amplify.ent and the lang.ent alias will get updated to point to lang.graft. Again, no guarantees which one. This is a simple example and users should expect these to work, but as you can imagine, do this a number of times throughout an alias heavy DAG, and things may become rather indeterministic and thus hazardous.

hazardous sensitive

We discussed this one already, but for completeness:

  hazardous string apiKey = "secret_1975718278757971981018830981307557_ACF4";

It is bad practice to have sensitive literal values in our code, but during prototyping it can be defensible.

hazardous typecast

We get ahead on ourselves a little bit with this one. Hazardous typecast cut into the Merg-E numeric type system and virtual numeric types that we will get into later.

  inert whole8 myval = hazardous typecast whole8 ( 128 + 1 );

While 128 + 1 should resolve to 129 and should fit into a whole8 quite comfortably as whole8 can hold the value 0 upto 255, that is not how Merg-E treats numeric literals. 128 is treated like a virtual 8 bit number. 1 is treated as a virtual 1 bit number. Assigning a 1 bit number to an 8 bit number might overflow and in the generic case we need at least 9 bits to avoid that, and as there are no concrete 9 bit whole number types in Merg-E without the typecast 128 + 1 would resolve to a whole16.

Here the hazardous typecast tells the toolchain that the programmer has looked at the numbers and has determined that 128 + 1 is going to fit into an 8 bit whole number.

hazardous blocker

Normally it is bad practice to await anything inside of a lock. Awaiting causes dependencies on at least other scheduling and possibly on asynchonous IO events, keeping the lock active for a potentially long time, what is bad. There are however rare circumstances where there are no other options and we need to await inside of a lock. The toolchain will refuse blocker instantiation or usage within a lock unless it is decalred as hazardous.

reentrant mutable function report (count whole16)::{
  cout;
  endl;
  cerr;
  }{
    lock(cout , cerr) {
      mutable hazardous blocker flushed;
      %if ok_count < 1 {
        cerr "ERROR: negative count" endl;
        flushed += cerr.flush();
        await.all flushed;
        };
      cout "counted " ok_count " prime numbers" endl;
      };
    };

It is important to note that an active hazardous blocker will put the scheduler in panic mode. The scheduler may abandon fair scheduling if any hazardous blocker has the potential of being resolved what might lead to temporary scheduling starvation of non hazardous tasks. As such hazardous blockers should be used sparingly and with the utmost care.

hazardous merge

This is a weird one out in the bunch. The hazardous modifier in a merge is meant to convey transitivity and define a proxy point for use of hazardous within a module. Also note the new syntax for modifier that is required here.

  hazardous modifier::<blocker> reentrant mutable merge utils.is_prime as is_prime(x int)::{
    ok_count;
      }{
        max_prime;
        }@[range_error];

The above code says: With this hazardous merge, the programmer tells us that she believes that utils.is_prime can be trusted to use a hazardous blocker.

Hazardous fidelity promotion

One more time we get ahead of ourselves by looking into the numeric type system. Merg-E is pretty serious about fidelity of 

  inert nodim rquantity16 pi = 3.140635;
  inert whole8 one = 1;
  inert nodim rquantity32 pi2 = pi + one hazardous;

If we multiply the low fidelity (16 bit floating point number with no dimensions) number pi with the whole number one, the result of the "+" operation will normally be another low fidelity 16 bit float that we can't assign to a higher fidelity version of pi. By using the hazardous modifier on the "+" operator, the result becomes castable to a higher fidelity constant.

While this is generally considered as non-idiomatic, sometimes APIs require it's usage.

--trustmebro

None of the above uses of hazardous will by default work with the toolchain. We need to tell the tools in the toolchain what uses of hazardous we are allowing in the current specific phase of our development cycle. We do this using the --trustmebro flag. For example:

merge-bones /home/me/my/app/dir/ --trustmebro=merge:blocker:typecast,prune

In this example the application is allowed to use hazardous prunes and hazardous merge, and a hazardour merge is allowed to grant a merged inner function to use hazardous blockers and/or hazardous typecasts.

The harsh critic: The Edsger linter and static analyzer

Before getting permission from Mark Miller for the usage of the name Merg-E for the language, and before hazardous/--trustmebro made it into the language spec, the fallback name and long time working name of the project was Edsger after my compatriot and idol Professor Edsger Dijkstra. While Merg-E in it's current hazardous iteration would no longer be fitting of the name edsger, we kept the name as part of the project as the name for a particularly pedantic linter and static analysis tool that is currently in early stages of development. While not part of the core toolchain that we will discuss later, it is recommended that developers have a look at the edsger output when run on their code. Deep discussion of the static analysis and linting features planned for edsger falls outside of the scope of this post, but mentioning its future existence is important.

Aliases for hazardous

While aliases have no influence on the base toolchain outputs or the runtime, some may influence Edsger's assessment of the code. Two of these aliases are provisional and heterodox. Both are language-defined aliases for hazardous, but they communicate different programmer intent and are therefore useful to static analysis and human readers.

So while the base Merg-E toolchain and runtime treat provisional and heterodox as identical to hazardous, and while --trustmebro handles them all exactly the same way, these aliases exist specifically to communicate programmer intent to the edsger linter and human reviewers.

Some uses of hazardous are expected to be temporary: a prototype may contain a hazardous construct that is intended to be replaced or properly resolved later. Other uses are deliberate deviations from the preferred Merg-E idiom and may be intended to remain permanently.

For the latter class, it is considered idiomatic to use heterodox instead of hazardous. This explicitly tells the reader that the deviation is intentional rather than merely provisional.

For provisional hazards, there are two equivalent idioms. The plain hazardous form is implicitly provisional:

  hazardous string apiKey = "...";

Alternatively, the programmer can make that provisional status explicit:

  provisional string apiKey = "...";

Neither form changes the semantics of the hazardous construct or the authority granted by --trustmebro. The distinction is information for the programmer, reviewer and tools such as edsger.

The idiomatic choice is to be consistent within a source file: either leave provisional hazards as plain hazardous, thereby treating them as implicitly provisional, or mark them all explicitly as provisional. Mixing the implicit and explicit forms of provisional hazards in the same source file is considered non-idiomatic.

The use of heterodox is different: it should be used whenever the programmer considers the hazardous construct an intentional and potentially permanent departure from the non-hazardous Merg-E idiom.

  inert whole8 myval = heterodox typecast whole8 ( 128 + 1 );

Edsger-only prefix characters.

The main pipeline (bones/muscle/skin/cyborg/runtime) will ignore the special prefix characters % and @ on resolving non-canonical alphanumeric tokens, but they are part of idiomatic Merg-E, make Merg-E code more readable, and are strictly checked for idiomatic usage by edsger.

  • The % character as alphanumeric token prefix denotes: This is a skin-macro or skin-macro alias.
  • The @ character as alphanumeric token prefix denotes: This is a modifier or modifier alias.
  @inert whole8 myval = @heterodox typecast whole8 ( 128 + 1 );

The above is an example showing idiomatic use of @ to denote modifiers. This form makes the resulting code slightly more easy to read because modifiers stand out.

   %while (x < 17) {
     ...
     };

The above example shows idiomatic use of % to denote skin-macros. In Merg-E, many flow-contol constructs such as %if, %while, %take, etc are thin abstractions on top of actorcitos and structural iterators. The idiomatic use of % to prefix these constructs makes it clear that skin will dissolve this construct into runtime-appropriate building blocks.

Part 2.

This is as much as HIVE allows us to post in a single post. In part 2 we continue this language-architectural in-a-nut-shell.

Comments

No comments yet — be the first.