A lot of what I've shared recently has been about Holo hosting -- getting all the bits really robust so they handle both 'happy path' user interactions (everything is working as expected) and 'sad path' ones (HoloPort goes down, login fails, etc). But I haven't spent much time talking about how Holochain itself is improving -- and when I have, it's usually been about bugfixes to support the aforementioned Holo work.

But behind the scenes, Holochain has been getting some big upgrades to some of its parts. This is mainly to make it faster -- fast enough for the next stages of Elemental Chat testing. (And of course, Elemental Chat testing is really about Holo hosting and Holochain.)

LMDB to SQLite: the road to infinite scalability

A lot of peer-to-peer networks, from decentralised social networks like Secure Scuttlebutt to blockchains like Bitcoin, are built on something called 'gossip protocol'. It's a simple protocol, and it works like it sounds like it would work: I know a bunch of people (or rather, computers), and when one of my friends shares something with me I share it with everyone else I know. With this protocol, messages spread rapidly through networks without a central authority.

And it also creates a lot of network traffic. That might be fine for blockchains, in which every machine is online most of the time and needs to know about everything anyway. But for networks that power everyday applications, you want to be respectful of your users' bandwidth -- and you can't expect their machines to be on all the time either. (Even blockchains tweak gossip to make it more efficient.)

Holochain is based on the idea of 'neighbourhoods' of authority -- each piece of data gets an address, and it's only sent to a group of machines that have taken responsibility for the range of addresses that cover its address. That means traditional gossip is too chatty -- as an authority, you need to constantly make decisions about what to share with your neighbours, given that they aren't responsible for the exact same range that you are.

It gets more complicated when machines are turning on and off all the time (which happens a lot when they're just users' devices). A machine that's been offline for a while will want to get up to date with the latest gossip, and will ask its peers for a digest of new stuff since they were last online.

This means you have to be able to ask your own database questions like "which of my neighbours' responsibility ranges start before, and end after, address 531993928?" and "what new data have I collected within my neighbour Alice's responsibility range 112513216 to 241661317 since yesterday morning at 08:37 UTC?" These sorts of questions, called range queries, are really hard for a key/value database like LMDB to answer, but they're really easy for a relational database like SQLite.

Because all data in Holochain is just keys (addresses) and values attached to those keys, LMDB initially made sense -- it was much faster than any relational database out there when it came to key/value lookups, as well as being gentle on CPU and RAM. But as the core developers started looking at sharding, it became obvious that the large amount of range queries would hurt performance unacceptably if they stuck with LMDB.

As a side bonus, SQLite also makes it possible for us to encrypt source chain and DHT data at rest using SQLCipher. (Note: this doesn't prevent you from reading your peers' published data -- that's not how Holochain works. But it does prevent an attacker from stealing it if they gain access to your computer.)

Other gossip upgrades

As mentioned, when machines come back online after being offline for a while, the catching-up process has been pretty network-heavy. Now when it happens, machines compare their lists of held data using Bloom filters. This technique results in more back-and-forth messages between nodes but less data being exchanged overall. This is already speeding up syncing, even though sharding is yet to be introduced.

New versions of Wasmer and Rust

Wasmer, the WebAssembly engine we're using to run the 'rules' of a Holochain app, is now at version 1.0, so we've upgraded it. It comes with better memory management, a faster compiler and runtime, and a bunch of bug fixes. It also lets us compile and run Holochain on ARM CPUs like the HoloPort Nano.

Wasmer 1.0 will also let us implement compute metering for Holo hosting, as well as runtime budgets to prevent infinite loops from chewing up all of a host's CPU capacity.

A new version of the Rust programming language has been released as well, so we've upgraded Holochain to use that too.

As a side effect, macOS developers are now first-class citizens again -- these updates plus some bugfixes have resolved the macOS build issues for Holonix, our development environment.

What's next?

Wasmer 1.0 and Rust 1.52 have already been integrated into Holochain, and they'll be rolled out along with the SQLite update. Expect to see faster install times soon too; we're researching automated build servers for Holonix so you don't have to wait an hour for everything to compile!

The work of integrating SQLite is done, and the slower performance of key/value lookups is more than offset by the faster performance of range queries. It's currently being tested on a single HoloPort with an automated Elemental Chat test involving 20 hosted users and one self-hosted instance, which has revealed a bug related to clogged processing queues. If there are no other show-stopper bugs, it'll replace the current LMDB-based Holochain on all HoloPorts soon.

Now that this work is almost ready, the core devs have been able to work on sharding the DHT. So far it's been 'full-sync', which means everybody stores everything, a lot like a blockchain. This is good for redundancy but bad for performance (although community testers have been consistently talking about how fast Elemental Chat already is). So far they've completed about half of the tasks involved in sharding.

Sharding has always been part of our promise to make dApps fast and scalable, so I'm excited to see this happening.

Reminder: Register for Dev Camp!

The next community-led Dev Camp is starting at the end of June. The project you'll be working on, a game called Tragedy of the Holocommons, will help you understand how to design an app that works well without a single source of truth. I think it's a brilliant idea.

Dev Camps are still free, and they’re run by a fantastic and knowledgeable group of volunteers.

The Holochain developer community is growing, along with the ecosystem of learning materials and open-source libraries, tools, and apps. If you've been holding back, waiting for the ecosystem to mature, it might be time to check out the forum and GitHub and get involved. These are my favourite resources; please add your own in the comments!

Cover photo by Julian Hochgesang on Unsplash

Install Holochain Learn about Holochain Join the Discussion