Hello,

Two months ago I posted about Gopherd, the second consensus implementation for Hive that I've been building from scratch in Go. Back then it could replay up to 45 million blocks and match hived bit for bit, and I said what was left was "a mountain of edge cases and operational work".

Well, I have something to show you :

That block was built, signed and broadcast by Gopherd. The network accepted it, the next witness built on top of it, and it's part of the history of the chain, just like any other block.

It's the first block in Hive's history produced by something that isn't hived.

So as of August 10th, Hive is a multi client chain* !!! _{^{* Not really gopherd is still in the early stages and there's lots to do before it's production capable}}

Why are you so excited about a single block ?

A block is 3 seconds of chain. This one took about four months of work.

Basically to produce a valid block you have to be right about everything, all at once. You need the complete chain state (every post, vote, comment, transfer, power up, witness vote, burned HIVE, proxies, escrows, rc, hard fork rules, the migration from steem to hive etc etc) computed exactly like every other node from genesis to now. You need to be able to communicate with other hived nodes, know when to produce, manage the mempool, select transactions, apply them, get the merkle root right, sign the header with the right key, and hand the thing to the network in the exact shape the protocol expects. And that's just scratching the surface.

All the nodes operate a very precise ballet dance and you need to match the pace and movements otherwise you get thrown out.

Actually this is an included block but it's not the "finished" version yet, you'll notice that the block has no transactions, only vops, because at the time my mempool implementation was flawed so gopherd didn't include any "real" transactions. Since then I've fixed it and now it can produce full blocks with actual transactions

So it's a single block but it's a big milestone nonetheless. "a small block for man, a big block for hivekind" (I'm joking).

What it proves is that two completely independent codebases that share zero lines of code now agree on what the Hive protocol is.

What does this actually buy us ?

Right now basically every node on Hive runs the same C++ codebase, which means every node has the same bugs. When there's a latent consensus bug in hived there is nothing anywhere in the network that is in a position to disagree with it and carry on.

We've seen where that goes. At HF20 in september 2018 (back when we were still STEEM) a constant got changed that shouldn't have been, the chain forked, and when witnesses started reverting to a stable build that exposed a SECOND bug in the fork database logic which halted the minority fork too. Chain stopped. Same code everywhere, same failure everywhere, and nobody left running something different that could have kept producing. A chain halt for a blockchain is almost the worst case scenario (worse being loss of funds ofc). It significantly weakens investor trust and overall hurts all the users. What if I had one million HIVE in a conversion request that was ment to be executed in three days, the chain halts for a week, the price for HIVE changes significantly and then my conversion request ends up wildly different and I lose thousands.

Now let me be precise about what a second client fixes and what it doesn't, because I've seen this oversold in marketing pitches and I'd rather not do that.

Hive has 21 witness slots per round, and irreversibility needs 75% agreement, so 16 of those 21. Which means :

If a bug takes out the hived nodes and some witnesses are running Gopherd, those witnesses keep producing their slots. The chain keeps moving, degraded, instead of flatlining. Even a handful of slots is the difference between "slow" and "dead".
But for the chain to keep FINALIZING, for transactions to actually become irreversible, you need 16 of 21. So the real protection isn't "a couple of people run the other client", it needs a meaningful chunk of the top 20.

That's the honest version. A second client turns a total halt into a degraded chain, and the more witnesses run it the closer we get to genuinely not caring which client has the bug that day.

It also cuts both ways, and I have a very good example of that further down.

The other reasons I think multi client is worth it

It turns the spec into an actual spec. Right now the specification of Hive IS the C++ code, including all its quirks and its "this looks like a bug but it's load bearing now" behavior. Writing a second implementation forces every implicit rule to become explicit, because you can't match behavior you haven't understood. I've got a folder of notes now on how Hive actually behaves that didn't exist written down anywhere (except in c++ I guess).

It allows us to experiment and have a different approach to the same protocol Building gopherd from the ground up allows me to test a lot of things and experiment, like the way I manage state and sync is different than hived's. As an example, Ethereum is very big on its multi client roadmap and supports 7 different clients that all have different approaches to the same protocol which led to innovation (I.e state pruning was pioneered by parity in 2016 vastly reducing disk costs and most other clients followed suit later on). I don't think hive is big enough to warrant 7 but I do think having multiple client is worthwhile.

It's a permanent fuzz test. Two implementations that have to agree block for block will keep finding each other's mistakes forever, not just at launch. Every hardfork from here on gets checked twice by construction.

It spreads the bus factor. hived is written with a lot of very specialised C++ and the set of people who can safely change it is quite small. A second implementation in a more accessible language widens the pool of people who can reason about consensus.

The part where I broke the network

Ok so this is the bit I owe everyone a proper writeup of, and it's also the best argument for this project so far.

About a month ago, as I was working on p2p sync Gopherd caused a significant forking incident on mainnet. Nodes fell out of sync, participation dropped to 61%, and there was a second order effect that broke a bunch of API nodes meaning that for most users, the chain was unusable (no api nodes means you can't send transactions or get posts from a front end).

It's what triggered the 1.28.7 release. That was my fault and I'm sorry for the mess it caused witnesses and node operators.

The mechanism is genuinely interesting though, and it was a latent weakness that had been sitting in hived for years.

Basically, when a node serves a block that's considered too old, hived disconnects it. That protection exists because serving up ancient blocks is the kind of thing a malicious node does. Normally this is harmless : the node gets kicked, reconnects, negotiates, finds out it's out of sync and goes into sync mode, and in sync mode nodes don't kick each other over old blocks because being behind is the whole point of sync mode. It heals in a few seconds.

Two things made that fragile:

First, one block irreversibility (OBI) shrank the definition of "too old" a lot, a block can now be too old at three seconds because we usually only have the one block of slack.
Second, blocks travel two ways, through syncing and through inventory advertisement, and the strict kick applies to the inventory path.

Gopherd was in sync and advertising blocks, but because the computer it ran on was too cpu starved, by the time it served the block, the blockchain would already be onto the next one. So it would serve old blocks while presenting itself as in sync. So it took the strict path instead of the forgiving one.

And here's the bit that turned one bad node into a network event : the kick was too slow. Before a peer finished deciding that Gopherd was bad, it had already relayed those old blocks onward to its own peers. So that peer got kicked by its peers. And those peers got kicked in turn. One misbehaving node produced a cascade of perfectly healthy nodes disconnecting each other, over and over, because I kept reconnecting and feeding it again.

The fix in 1.28.7 was to stop treating those blocks as invalid and treat them as unlinkable instead. That sounds like a technicality but it's the whole thing : "invalid" means disconnect the sender and everyone else advertising that block, "unlinkable" routes it into the fork reconciliation path, which is just sync mode, the forgiving one I described above. So a routine micro fork stops turning into a fork storm.

Then the forking exposed something else. Under sustained forking a latent bug in HAF's fork handling surfaced. That one got introduced at some point in HAF's history and had simply never been hit, because Hive doesn't normally fork enough to reach it. That's the one that actually hurt users since it degraded API nodes, and it left some HAF databases in a poisoned state on top of that. Both got fixed fast, and the fix even recovers already affected HAF nodes in place instead of forcing a resync from scratch. Credit to @blocktrades and the team, for a very quick turnaround.

So to sum up : my node caused an outage, and it also flushed out two real bugs that all our testing on hived had never triggered, one of them years old. Which is exactly the argument for a second implementation, It's just delievered a lot more loudly than I intended.

To quote @gtg "while that node was a cause it wasn't their fault, it found conditions that we were unable to find in synthetic tests, even slowloris-grade attacks or other ideas of straving peers so yeah, good job ;-)"

If you want to learn more about this, @blocktrades gives a very good rundown in the last core dev meeting

What's been built since the june announcement

Roughly 250 commits since the last post. The highlights :

The replay reached chain head. Not 45M blocks, all of them, matching hived, and then catching up to the live tip and keeping in sync.
Live P2P sync. Gopherd connects to mainnet seed nodes, syncs to head, serves blocks to hived peers and holds the tip. It's a real participant in the gossip network now, not a replay tool.
Fork switching, so the node handles tip reorgs instead of wedging on the first micro fork it sees.
Block production, in three stages : a dry run that builds and signs a block then throws it away (so it can be exercised on mainnet without touching anything), then key gated real production that refuses to produce unless the chain's registered signing key actually matches the key it holds, then the real thing.
A block assembly harness that builds a block every slot and diffs it against the block the real witness produced for that same slot. This has been by far the most useful tool in the project.
The API layer : 64 read methods across database_api, block_api, condenser_api, account_by_key, tx_status, plus a full rc_api on top of the ported RC subsystem.
Snapshots and tiered storage, so you can warm start from a checkpoint in minutes instead of replaying from genesis every time.
A pile of consensus fixes found by diffing against hived : HF26/27 gates, governance expiry spread, delayed voting, vesting pool drift, and a fun one where an IPv4 byte order bug meant peer discovery had never actually worked and we'd been dialling complete strangers for weeks.

And a lot of very unglamorous operational work : memory bounding, cold tier eviction, restart behaviour, disk pressure, all the stuff that decides whether a node survives a night unattended.

Please don't run this as a witness

I want to be really clear here because I know some of you are exactly adventurous enough to try.

Do NOT run Gopherd as a production witness. I also do not run it as a witness, I only set my keys to it to test producing one block and then switch back to my hived nodes. it is not ready. I'm still actively developing it and I'm still finding consensus edge cases. It's very much fragile. There will be a version I'm comfortable handing to adventurous beta testers and I'll say so when there is (and FYI it will be a long time in the future as I want to be extremely safe). Until then treat this as a research project that occasionally produces a real block. I might create a second witness that burns all post rewards but that exists in the network so that I can test liveliness without affecting the reliability expected of a top 21 witness.

Thank you for reading, if you like my work please consider voting me for my prop... Well about that

About my proposal

I'm cutting my proposal in half, from 300 HBD a day to 150 HBD a day.

I've been working on Hive for free for the last three months. My proposal hasn't been funded and I kept working anyway, because I care about this chain and want to see it succeed. But I can't keep working for free forever. And eventually I will need to get another job, which will take a lot of time, time that I can't spend on HIVE. So I sat down with my actual finances and worked out the lowest number where I can justify doing this. and that number is 150 HBD a day. It's a drastic pay cut for the same work, and I'd rather take the cut and keep building than ask for more and stop.

Consider it my commitment to the community. Everything I promised in the original proposal still stands, the community and Hivemind work, the core development, the hardfork contributions, the core dev meetings. And on top of that I'll be bringing Gopherd to life, supporting it and maintaining it as a real second client for Hive.

I think this is important work. A chain with a single implementation is one bug away from a bad day, and I'd quite like Hive to not be that chain.

As per usual, if you like my work, feel free to vote me as witness or/and support my proposal : https://peakd.com/me/proposals/371

And if you've got questions on the technical side (or not), ask away in the comments, I'll answer all of them :)

@howo

Hive is now a multi client network, the forking incident I caused, and reducing my proposal to 150 HBD a day