#technology

What I am looking for from the witnesses

Introduction: What I am looking for from the witnesses

Yesterday's hardfork made me begin to rethink what I want to see in exchange for my witness votes. The Steem block chain is 2 1/2 years old now, and it's about time for its governance mechanisms to start maturing. I believe that one key to this maturity is the participation of witnesses.

Image Source: pixabay.com, License: CC0, Public Domain

Until yesterday, I have been casting votes for witnesses who I think do generally good things for the Steem ecosystem. After being locked out of the Steem blockchain yesterday for just about 24 hours, I am rethinking that methodology. Witnesses are handsomely rewarded for their role, and I am now thinking that community members should begin to expect a more active governance role from the witnesses.

I've worked in IT for a long time, and I know that no code is ever perfect. I also know that programmers and engineers can be overconfident, and they often need to have external discipline imposed on them. That's not intended as a criticism. It's just human nature. In my experience, it is true just about everywhere that you find cutting edge technology.

To be honest, I was ok with the "Hardfork Successful" headline in yesterday's post. I do think it was a success, but the reality is that the success was very expensive. Now that things are settling down, it's important to start learning the lessons that it can teach us. Future successes need to come at a much lower cost.

So, witnesses, I am looking for you to begin to apply that necessary external discipline. You are the board of directors of the Steem block chain - the most powerful group of stakeholders. Off the top of my head, here are some of the kinds of things that I would like to see before I cast my future witness votes.

1. When the block chain is functioning smoothly, your default position on accepting a hardfork should be "no".

Steemit, Inc. (or anyone who proposes a fork) should have to convince you - in formal public proposals, not vaguely worded and sporadic blog posts - that the hard fork will be good for your voters. You should look at their communications critically, and demand a high level of evidence and confidence that the hard fork is a move in the right direction.

Forks are good, and you should accept them when appropriate, but you should never "rubber stamp" them.

2. When you do agree to accept a hard fork, you should be prepared to back out the change quickly.

To me, insofar as possible, this means three things. First ensure that the forked code makes a provision for backing out after activation (without losing any transactions that don't depend on the new features). Second, that you should be prepared with redundant hardware, software, and/or configuration settings so that you can restore service under the old version at a moment's notice. And finally, before the fork you should have a checkpoint in mind at which time you plan to initiate a back-out. (and that time should be well under 24 hours)

3. You should audit the code prior to forking.

If you are not capable of auditing the code on your own, you should hire someone to do it for you. This hiring could be done individually, or in collaboration with other witnesses.

4. You should thoroughly test the code prior to launch.

There are certain things that should be part of standard test suites. These usually include things like outrageously small and large values, negative results, boundary conditions, and back-out capability.

On Steem, I'm learning from the past couple hard forks that a "boundary condition" which needs to be thoroughly explored is the time period of a week or so before and after the fork. Going forward, I will be looking for witnesses who announce publicly that they have thoroughly tested the entire transition period as well as the new features that the the fork is intended to deliver.

As with code audits, depending on your skills, you may want to hire professional consultants to perform this work.

Conclusion

Obviously, that is not a comprehensive list, but it's a starting point. While I continue to appreciate the things of a general nature that witnesses have been doing to improve the Steem ecosystem, that sort of generalized activity will no longer be enough to earn my witness vote. Witnesses are being rewarded substantially for their role, and my future votes will only go to witnesses who will aggressively take on a technology governance role for the block chain.

Quite some time ago, @ned said somewhere that the most important characteristic for a witness is the ability to audit the code. In my mind, hard fork 20 has substantially vindicated that statement.

I know that my stake is relatively small, but for whatever it's worth, nearly all of my witness votes are now officially up for grabs. Going forward, they will only be reassigned to people who I believe are willing to perform a Steem technology governance role and apply some change discipline during forks in order to tamp down any overconfidence that might exist on the part of the Steem development team.

45% of this post's rewards will go to benefit the Rustin Golden Knight's Marching Band.

The @rgkmb-unofficial account is an experimental account that is operated by Steve Palmer (@remlaps). The account's purpose is to demonstrate the viability of a fund-raising model that is based on use of the Steem block chain in the community. The account's goal for 2018 is to raise enough money to send one student on the Rustin Golden Knight Marching Band's trip from Pennsylvania to Florida in December for performances in the Citrus Parade and in Disney. For more information, see our introductory post: Introducing Steem to the Rustin Golden Knights Marching Band.

As a general rule, I up-vote comments that demonstrate "proof of reading".

Steve Palmer is an IT professional with three decades of professional experience in data communications and information systems. He holds a bachelor's degree in mathematics, a master's degree in computer science, and a master's degree in information systems and technology management. He has been awarded 3 US patents.

He is also a co-founder of the Steemit's Best Classical Music Facebook page, and the @classical-music steemit curation account.

Classical Music Logo by ivan.atman

Follow: @classical-music

Follow: @classical-radio

Comments · 7

@jesta(71)· 2892d
While I generally agree with the direction your post headed, I figured I'd take a few bits of it and leave some responses for clarity.

Starting at the top:

So, witnesses, I am looking for you to begin to apply that necessary external discipline. You are the board of directors of the Steem block chain - the most powerful group of stakeholders.

This first comment is a bit nitpicky, but I think it's an important distinction to make. I'm not entirely sure this is an apt metaphor. If we were actually the board of directors, we'd be setting the direction for development and giving direction to the CEO of the company (like a traditional board does). We don't have that power unfortunately, as much as I wish that was the case. In a traditional company, events such as this would lead to an overhaul of the companies internals, which is completely out of our control.

People often overestimate the powers witnesses have over the direction, when in reality we are merely the gatekeepers that get to say yea/nay to proposed changes.

Regarding your 4 bullet points, I'm going to address them one by one in relation to HF20. Let's start with #1:

#1 - When the block chain is functioning smoothly, your default position on accepting a hardfork should be "no".

I'd argue that the blockchain pre-hf20 wasn't functioning smoothly, and that change was needed. We had a system of account creation and resource consumption which was not sustainable (for many reasons). These reasons include the cost of account creation, it's dependency on Steemit Inc to perform (or users pay for themselves), the requirements of delegations to allow participation, etc. I could write an entire post about how Steem has been fundamentally hamstrung by these factors - but the point is that I don't think many witnesses believed the blockchain was actually functioning smoothly. This update was sold as a solution to these issues, which in theory sounded great.

In hindsight, compared to today, it was functioning "more smoothly", but it did have it's problems that needed addressed.

Steemit, Inc. (or anyone who proposes a fork) should have to convince you - in formal public proposals, not vaguely worded and sporadic blog posts - that the hard fork will be good for your voters. You should look at their communications critically, and demand a high level of evidence and confidence that the hard fork is a move in the right direction.

I completely agree with this, but it's never been the case. I don't think you're going to find a witness who thinks the communication from the devs is good. We have been stressing this for years now - with improvement, but too little IMO.

#2 - When you do agree to accept a hard fork, you should be prepared to back out the change quickly.

I agree with this sentiment, but unfortunately it's not that simple, especially with this HF. Details aside, one of the major blockers here is that all transactions from Steemit.com flow through the API servers that Steemit Inc operates, and regardless of what version the witnesses are operating the resource requirements would be enforced on that controlled infrastructure.

For what it's worth - every witness to my knowledge had (or still has) a HF19 node running ready to take over. We, from the witness side, were ready to swap production (and likely still are), but it will require more than just action from the witnesses - it would need to be in collaboration with Steemit Inc to change their infrastructure.

#3 - You should audit the code prior to forking.

@ats-david covered this pretty well in his comments - but the code was reasonably reviewed IMO for what it was. The audit-ability of the code itself is the core concern for me (as well as many others) and we have long asked for more regular/frequent/smaller updates.

The hard part here is splitting the intent of the code vs the results of it's release. I don't think an audit of the actual code itself would have specifically prevented any of the issues we're seeing today - but maybe a longer/bigger/more-real testnet might have (which is another area that's improving, but could use more).

#4 - You should thoroughly test the code prior to launch.

There are certain things that should be part of standard test suites. These usually include things like outrageously small and large values, negative results, boundary conditions, and back-out capability.

This follows up nicely with the comments of testnets I mention above. Steem isn't a standalone application that just runs and can have 100% of tests performed on it - it's a massive system where the actions of each user affects all others, rippling through the overall application state of the blockchain. Without real user activity under real conditions, it will never be 100% covered simply by testing.

For instance, accounts of many sizes were tested on the testnet itself, but without the real world activity (and real users doing things), the ripples of resource constraints didn't appear on the testnets and things that are failing today in live worked fine.

It's a hard nut to crack, but I do agree though that this needs to be improved. This was the first HF (I think?) that a real testnet has actually be deployed, so at least there's a bit of progress on that front.

Just to close this out - you're absolutely right that everything needs to be improved upon, and this will be a lesson to us all. As jaded as I am most of the time, I tried to point out a few of the things that have improved since the last HF, and have hope that it will continue to improve over time.

There's a limit though as to how far these events can be simulated and anticipated, especially from the witness perspective.
@kevinwong(77)· 2893d
Good thoughts and agreed on all points. I'm all for a diversity of characters of any background and interest in place of witnessing, but a good portion of the funds in top 20 should be prioritized for a dedicated technical department if one does not have what it takes. But on balance, there's an odd misalignment of economic incentives on the platform which ought to be fixed so that witnesses can also function better (I'll leave this comment rant-free!)
@trufflepig(64)· 2893d
Congratulations! Your post has been selected as a daily Steemit truffle! It is listed on rank 22 of all contributions awarded today. You can find the TOP DAILY TRUFFLE PICKS HERE.

I upvoted your contribution because to my mind your post is at least 6 SBD worth and should receive 128 votes. It's now up to the lovely Steemit community to make this come true.

I am TrufflePig, an Artificial Intelligence Bot that helps minnows and content curators using Machine Learning. If you are curious how I select content, you can find an explanation here!

Have a nice day and sincerely yours, TrufflePig
@ats-david(75)· 2893d
When the block chain is functioning smoothly, your default position on accepting a hardfork should be "no".

This is my default position. And since STINC has demonstrated incompetence in the past and continues to bundle several large changes per hard fork (along with many smaller ones), it will remain my default position.

You should audit the code prior to forking.

Since STINC chooses to package massive amounts of code into each hard fork, this is a big ask from anyone auditing the proposed changes to the protocols. The first step should be to reduce the size and number of protocol changes, and to reject any proposals if...
1. There is no actual problem in need of a “fix.” and
2. Multiple large protocol changes are introduced in a single proposal.
If the changes are necessary and presented in the smallest package possible, then the code should be reviewed. And every witness in the top-20 should either have the required knowledge/experience to review it or hire someone who can.

After reviewing the code, it should be fully tested - and only then should it be proposed as a hard fork. So I agree with you on that as well.

Witnesses are being rewarded substantially for their role...

This is certainly true for the top-20, and somewhat true for the top 30-40. After that, the rewards aren’t much after server costs and time spent (especially when needing to fix things after STINC’s mistakes). Many witnesses below 60-70 are probably operating at around even or at a loss.

If anyone isn’t happy with how these hard forks are being accepted, feel free to remove your votes from those top witnesses and find better alternatives to take their place.

I can promise you this:

If I found myself in the top-20, I would certainly have my team reviewing and testing code, operating a full node, and engaging with the community. I already do the latter, but the others are cost prohibitive at #60 on the list.

If you want better witnesses, we’re out here. We just need the support. We hate seeing so much of our STEEM rewards going to waste on absent, unmotivated, and/or incompetent witnesses just as much as the rest of you. It’s frustrating...and sad. We can do so much better.
@tcpolymath(69)· 2893d
I'm willing to get paid in Steem to write unit tests for the next forks if that's something anyone wants. It doesn't seem like that happened adequately on this one.
@geke(70)· 2893d
This makes a lot of sense to me. I'll be following your lead, clearing out my witness votes and adopting your list.
@ervin-lemark(76)· 2893d
Very true and well written. I might follow your checkpoints, thank you.

I hope that somebody, anybody, will listen.