scrobble.life
HiveDevs

2nd update of 2025: Hive API stack v1.27.11 officially released

@blocktrades@blocktrades (80)479d

blocktrades update.png

Today we officially tagged the latest version of the Hive API node stack (v1.27.11).

We’ve been running various beta versions of the new stack for quite a while on api.hive.blog, and its been easy to see that it performs much better than the old stack running on most of the other Hive API nodes. The release version of the API is now accessible on https://api.syncad.com and we’ll switch to the release version of the stack on api.hive.blog tomorrow.

With the official release of the new stack, I expect most of the other nodes will be updating to it within the next week or so, and we should see higher performance across the ecosystem.

Here’s a quick summary on some of what the BlockTrades team has been working on since my last report. As usual, it’s not a complete list. Red items are links in gitlab to the actual work done.

Upgrading everything to build/run on Ubuntu 24

One of the main changes we made across the entire set of apps was to update our build/deployment environment from Ubuntu 22 to Ubuntu 24. This required more work than expected as this also involved an upgrade to a new version of Python, which is heavily used in our testing system and also by several of our development tools, requiring changes to that python code.

Hived: blockchain node software

We improved the snapshot/replay processing so that you can first resume from a snapshot, then replay any additional blocks you have in your current block log, instead of requiring you your node to re-sync those blocks.

Optimizations In progress

We’re currently finishing up a few long-planned performance improvements: 1) a fixed-size block memory allocator (https://gitlab.syncad.com/hive/hive/-/merge_requests/1525) and 2) a moving comment objects from memory into a rocksdb database.

While benchmarking the fixed-block allocator, we saw a 18% speedup in in-memory replay time and a much bigger speedup (23%) for disk-based replays. The new allocator also reduces memory usage by 1665 MB.

Moving comment objects from memory also drastically reduced the size of the statefile, which will make it easy for systems with relatively low amounts of memory to do “in-memory” replays. I’ll provide more details on this later after I’ve personally benchmarked the new code, but everything I’ve heard so far sounds quite impressive.

Upcoming work: enhancing transaction signing

We still plan an overhaul of the transaction signing system in hived. These changes will be included as part of the next hardfork and they are also tightly related to the support for “Lite Accounts” using a HAF API (my plan is to offer a similar signing feature set across the two systems to keep things simple). So the Lite Account API will probably be rolled out on a similar time frame.

HAF: framework for creating new Hive APIs and apps

Upcoming work: lightweight HAF servers

Plans to support an alternate “lightweight” version of HAF with pruned block data: https://gitlab.syncad.com/hive/haf/-/issues/277

HAfAH: account history API

We switched to using structured parameter/return value types for the new REST API. Only a few apps currently use the new API, and these apps (e.g. Denser and the HAF block explorer UI) have been upgraded to use the newer version of the API.

Hivemind: social media API

There was a fix to hivemind’s healthcheck due to an update to HAF’s schema: https://gitlab.syncad.com/hive/hivemind/-/merge_requests/867

Optimized notification cache processing to reduce block processing time

We optimized notification cache processing. Previously this code had to process the last 90 days worth of blocks (e.g. around 3m blocks) to generate notifications for users, so it consumed the vast majority of the time that hivemind needed to update on each block during live sync (it took around 500ms on a fast system). Now we incrementally update this cache on a block-by-block basis and it is 100x faster (around 5ms). Total time now for the hivemind indexer to process a block is down to a comfortable 50ms. There are a few minor issues with the new code, which we’ll resolve in a later release.

Redesigned follow-style tables to be more efficient

We also redesigned the tables and queries for managing follows, mutes, blacklists, etc. This not only reduced storage requirements, but more importantly, allows for the performance of queries related to these tables to scale well over time. In particular, functions that need to skip "muted" information should be much faster.

These changes were done in https://gitlab.syncad.com/hive/hivemind/-/merge_requests/863 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/869 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/872

Configurable timeout for long API calls

Hivemind also now has a configurable “timeout” on API calls that API node operators can set to auto-kill some pathological queries that might unnecessarily load their server. By default it is set to 5s which should be appropriate for most current servers I think. Very fast servers may consider lowering this value and very slow servers may want to increase it.

Balance tracker API: tracks token balance histories for accounts

  • Similar to HAFAH, the REST API was modified to support structured parameter and return types.
  • We added daily, monthly, and yearly aggregation data for coin balances.
  • Further speed ups to sync time.
  • Added limits for APIs taking a page size
  • Added support for delegation processing
  • Track savings balance
  • New API for recurrent transfers

Reputation tracker: API for fetching account reputation

HAF Block Explorer

HAF API Node (Scripts for deploying and managing an API node)

For anyone who wants to run a Hive API node server, this is the place to start. This repo contains scripts for managing the required services using docker compose.

  • Fixes to assisted startup script: https://gitlab.syncad.com/hive/haf_api_node/-/merge_requests/89
  • Various fixes to healthchecks
  • There is a separate “hivemind_user” role that is used to allow for timeout of API-based queries (as opposed to indexing queries). As mentioned in the hivemind section, this timeout defaults to 5s.
  • We now use “haf” prefix by default instead of “haf-world” to shorten container names.
  • Tempfiles under 200 bytes aren’t logged to reduce log spam
  • More database tuning settings were made based on analysis using pg_gather. In particular, work_mem was reduced from 1024MB to 64MB, which should reduce the chance for an OOM condition on a heavily loaded server.

WAX API library for Hive apps

  • Support for external signature providers (like Keychain) in transaction creation process supported by Wax
  • Eliminated issues reported by dependabot service specific to dependency and code based vulnerabilities
  • Implemented support for MetaMask as signature provider extension. We are waiting for security audit verification (of dedicated MetaMask snap implementation supporting Hive integration) to make Hive officially supported by MetaMask. Also Hive has been included in https://github.com/satoshilabs/slips/blob/master/slip-0044.md
  • Improving error information available to applications when API calls fail. First step is mostly done in Hived repo: generation of constants representing specific FC_ASSERT instances. After that, exception classes in WAX will wrap the most common error cases and then expose them to Python/TS to simplify error processing (currently complex regexp parsing is required on the client side to detect some types of errors).
  • Improvements to support workerbee better (bot library).
  • First working version of Python implementation. API support is still in progress, but we expect to have our first prototype for automating generation of the API call definitions for Python from the swagger.json file (same as it currently works for TypeScript) by next week.

Hive Wallet MetaMask Snap

This is a hive wallet extension allowing you to sign transactions using keys derived from your MetaMask wallet.

We are currently preparing the library for an official security audit by improving project documentation and fixing issues after an internal review.

Hive Bridge

To make joining Hive more smoothly, we created a Hive Bridge service providing basic features such as signing, encrypting (can be used in bot authentication flows where given user need to confirm its authority by encrypting some provided buffer). The service is available at: https://auth.openhive.network

WorkerBee

This is a typescript library for automated Hive-related tasks (e.g. for writing bots that process incoming blocks). We recently made performance optimizations to workerbee to support large scale block processing scenarios where lots of previous blocks need to be fetched. Recent work has speed this up to 3x. We’re continuing to work on further optimizations to eliminate bottlenecks.

Generic Healthchecker UI component

We officially released the healthchecker UI component. This component can be installed into a Hive web app to allow a user to monitor the performance of API nodes available to them and control which API node is used. The HAF block explorer UI was also updated to use this component.

Denser

HiveSense: Semantic-based post searching using vector embeddings

HiveSense is a brand new HAF app to optionally allow for semantic searching of Hivemind data (i.e. Hive posts). This should solve a long-standing problem where it has been difficult for users to find older content, so it should be a very nice improvement to the ecosystem.

It works by running deep learning algorithms to generate vector embeddings for Hive posts. These can then be searched to identify posts that are semantically related (related by meaning rather by than by exactly matching words) to a user’s search term.

This project is still undergoing development and testing right now, but the code is available for public experimentation. The new repo is here: https://gitlab.syncad.com/hive/hivesense

Comments · 27

  • @urun(69)· 92d

    Very cool updates @blocktrades . Is it possible to somehow ( even if its outsourced to a EVM chain) to get smart wallets. They could pay to play ( pay hive to transact).

    Like log in with a ETH wallet, use the ETH wallet on Hive ( sign it). So it can hold onchain Hive and to transact pay in Hive or HBD.

    Mechanic:

    EVM wallet, signs transaction ( no ETH cost) to verify wallet. EVM wallet data gets imported on Hive, ready to play.

    Some way to scale apps on hive an alternative to Hive wallets.

    The Mechanic itself could be RC pool ( users could delegate) and gets priced by demand ( this way RC gets $ value and pays yield). Or some Onchain mechanic i did not think off ( like a chain fee additional to RC).

    Something like that would be so cool, EVM users can use their wallet for anything on hive + can later make a account if they want.

    What do you think about something like that? I think it should be doable.

    Revenue stream + scaling.

    Testing should be easy on testnet + all vibe code if the idea is practical or not.

  • @kingsleyy(75)· 416d

    Hello, good day @blocktrades,

    Is there a way to get help from this account/project?

    Okay, from the look of things, it seems almost impossible to do that because it’s obvious this account is meant for a project. However, there’s no harm in trying, as I’ve seen this account being supportive of the growth of the Blockchain.

    We are humbly seeking support for The Comedy Club Community. It’s a newly created community designed to fill in the missing piece of laughter on the Blockchain. The main aim is exactly that—with no external intentions.

    To make this soliciting request brief, Please help us in any capacity—perhaps through HP delegation or by extending your hand in curation support for entries to our contests, under your terms and conditions.

    Sorry, I know this is supposed to be a private chat, but I thought it would be better (as an attempt) to make this request here. Maybe we could continue on Discord if you're okay with that. Also, kindly forgive my poor manner of making this request.

    Many thanks!🙌

  • @raymondelaparra(69)· 454d

    Congratulations @blocktrades

  • @sg-gardener(48)· 463d

    Congratulations for 1000 vote .. sir please help me //Build a whales 🐋 why unvote me

  • @sg-gardener(48)· 465d

    Sir your journey is so smooth for us

  • @brightest(41)· 475d

    Great development down here 👍😍

  • @danzocal(60)· 478d

    !PIZZA

  • @solominer(82)· 478d

    We have well over a hundred active Hive witnesses, but only around 10 active API nodes.

    Should we have more API nodes?

  • @driipam(64)· 478d

    Such cool developments happening! Thanks :)!

  • @jarvie(75)· 478d

    I wanted to understand this post in a way that was a bit easier for me and others so I asked Gemini for some help... figure this could help someone else so i'm pasting it here: (Obviously if there are any misunderstandings feel free to correct)

    Okay, Hive users, let's break down this tech update from Blocktrades! Think of this as upgrading the engines and tools that make Hive work behind the scenes.

    Here’s the lowdown on what Hive API stack v1.27.11 means for you:

    The Big News: Hive Just Got Faster!

    • The main point is a new, improved version of the software (v1.27.11) that powers many of the Hive websites and apps you use (like PeakD, Ecency, Liketu, Splinterlands, etc.).
    • Blocktrades has been testing it, and it's noticeably faster than the older versions most services currently run.
    • You can already experience this speed boost on api.hive.blog (switching soon) and api.syncad.com.
    • Expect most Hive apps and websites to feel snappier in the coming weeks as other node operators (the people running the servers for Hive apps) upgrade to this new version.

    What Else is Cooking? (Key Improvements)

    1. Smoother Performance & Less Hiccups:

      • Faster Notifications: A major bottleneck in getting your notifications quickly has been fixed. It's now 100 times faster on the backend! This means notifications should appear much more promptly in apps that use Hivemind (most social front-ends).
      • Faster Feeds (Especially with Mutes/Blacklists): They've improved how follows, mutes, and blacklists are handled. This should make loading feeds and filtering content (like hiding muted posts) much quicker and more efficient.
      • More Stable Nodes: Several technical tweaks (like better memory management, database tuning, and timeouts for slow queries) mean the servers running Hive apps should be more stable and less likely to crash or slow down under heavy load.

    2. Better Ways to Find Content (Coming Soon!):

      • Smarter Search (HiveSense): This is exciting! They're building a new tool called HiveSense that uses AI (vector embeddings) to understand the meaning of posts. Instead of just searching for exact keywords, you'll be able to search for topics or ideas, making it much easier to find relevant older content. This could be a game-changer for discovering posts!

    3. Wallet and Account Improvements:

      • MetaMask Integration Coming: Work is well underway to let you use your MetaMask wallet to sign Hive transactions via a "MetaMask Snap." It's waiting for a security audit, but this could make using Hive even easier for those familiar with MetaMask.
      • Better Balance Tracking: The system that tracks your token balances is getting upgrades, including daily/monthly/yearly summaries and support for tracking savings and recurring transfers. Apps using this might be able to show you more detailed info.
      • Easier App Connections: Improvements to how apps talk to wallets (like Keychain) and handle errors.

    4. Behind-the-Scenes Foundation Work:

      • Modernized Systems: Everything's been updated to run on the latest operating systems (Ubuntu 24) and software versions (Python), keeping Hive up-to-date.
      • Faster Node Syncing: Improvements make it quicker for node operators to start or restart their servers, meaning less downtime and faster recovery if issues occur.
      • Lite Accounts & Better Signing: Future work is planned to improve how transactions are signed and introduce "Lite Accounts," making things potentially simpler or more flexible down the road.

    In a Nutshell:

    This update is all about speed, efficiency, and building for the future. You should start noticing faster apps and websites soon. Big improvements like smarter search and MetaMask integration are on the horizon. The Blocktrades team (and other Hive developers) are working hard under the hood to make your Hive experience better!

  • @steevc(80)· 479d

    It's always good to hear of optimisations as that ought to help Hive scale up. Hive has been pretty stable for a long time now and that is vital. I know you do lots of testing, so I hope that continues.

    Keep up the good work.

  • @arshadkhan4421(62)· 479d

    Great efforts and progressive development.

  • @aftabirshad(56)· 479d

    You have really done a great job, you should be given a separate crown for that.Believe me, this platform really needs people like you.

  • @luthis26(64)· 479d

    Wow! This is an incredibly comprehensive update. The dedication of the Blocktrades team to improving Hive is really impressive. Looking forward to seeing these changes propagate across the network. Thank you for the detailed report.

  • @pizzabot(60)· 479d

    PIZZA!

    $PIZZA slices delivered: @danzocal(2/10) tipped @blocktrades sirsmokesalot96 tipped blocktrades

    Come get MOONed!

  • @sirsmokesalot96(65)· 479d

    !PIZZA

  • @ssjsasha(74)· 479d

    one of the only ones who is still here from way back... you are a saint sir

  • @cuteboi(61)· 479d

    This is a massive improvement and good updates. Thank you for the effort put in achieving this. You deserve an applaud 👏💪

  • @hivebuzz(74)· 479d

    Congratulations @blocktrades! Your post has been a top performer on the Hive blockchain and you have been rewarded with this rare badge

    Post with the highest payout of the day.

    You can view your badges on your board and compare yourself to others in the Ranking If you no longer want to receive notifications, reply to this comment with the word STOP

    Check out our last posts:

    Hive Power Up Month Challenge - April 2025 Winners List
    Be ready for the May edition of the Hive Power Up Month!
    Hive Power Up Day - May 1st 2025
  • @christybliss(67)· 479d

    We are glad to learn about these developments. Thank you

  • @mfontom(68)· 479d

    Thank you for the update. It will help us greatly as we engage here

  • @andablackwidow(69)· 479d

    Adding my 2c to hived's Optimizations In progress.

    First one - pool allocators - in principle does not reduce memory usage (at least not at current stage). In fact it increases it somewhat, however due to less fragmentation actual usable memory increases, hence less allocation is reported. It is the same effect we could already get by loading snapshot, except now the benefit persists instead of degrading over time. I'm pretty curious about the performance increase, especially in combination with just finished undo session optimizations. As far as I know pool allocators are not yet applied to objects created for undo session, so there is still room for improvement (might even show up in performance of regular sync, not just in extreme traffic in live sync and in colony+queen cycle). Also we have detailed data on average amount of objects needed at any given time which might allow some tweaking of pool sizes for each individual index, which might also slightly improve overall performance still. Once the MR is included in develop further task opens up, but Marek wanted to first get the estimation of how much actual memory consumption reduction we can gain with it, especially considering that the biggest item is getting much smaller with second mentioned optimization - comment archive.

    As for "comment archive" optimization, drastic reduction of memory usage is obviously expected, but the first benchmarks look too good to be true. Even the extreme pessimistic scenarios are faster than current version, with most of comment accesses to archive in RocksDB instead of fresh comments inside memory multiindex (measured on big blocks with over 14k votes in each block). It is not entirely impossible, because RocksDB might be algorithmically faster even though each access to data is potentially much slower, but that would mean it makes sense to take a deep look into type of multiindex we are using, and not just for comments - more opportunities for optimizations :o)

  • @acidyo(84)· 479d

    Awesome to see these come to light!

    That last one I know is going to make life easier for many users and myself who always think of a post but give up even trying to search for it.

  • @ecoinstant(78)· 479d
  • @itsmemic(70)· 479d

    Been following Hive for a while now and this release feels like a huge leap forward. Thanks for keeping us updated with such detailed reports, makes techies like me feel part of the process!

  • @blocktrades(80)· 479d

    We benchmarked the new stack on a AMD 9950X3D with two 4TB T700 nvme drives and 64GB of RAM. HAF itself, using a RAM disk temporarily for the state_file replays in 13.34h. Next, replaying the apps in parallel takes 49h39m (block explorer takes 7h38m, reputation tracker takes 11h20m, and hivemind takes 49h39m). So total time to replay on this quite fast system is 13.34h+49.65=63h = 2days 15hours.

  • @spiritsurge(62)· 479d

    Great set of updates!! Obvious big fan of Denser and Wax.

    Some questions:

    a) For Hive Bridge, I see that it is not detecting Metamask extension. But rather it is calling for Metamask Flask in "Requesting Account Creation". Is there a documentation that we can look at for Hive Bridge or is it part of Metamask snap?

    b) Is the HealthChecker UI used in any site somewhere?

    Checking the other projects. Well done to the core dev team.