Two weeks ago we wrapped up our first public test of Holochain + the Holo hosting infrastructure, and in my opinion, it went wonderfully. For those of you who weren’t involved, we used a proof-of-concept app called Elemental Chat to collect insights on what was and wasn’t working. Five thousand of you signed up and eagerly attacked it, sending each other about 30,000 messages in English, Turkish, Dutch, French, Greek, and other languages.

(Note: I realise this isn’t really news anymore, and I apologise for the gap in Dev Pulse publication. My attention was unexpectedly taken away from work for a couple weeks.)

The most important finding was that Holochain is already working pretty well. In fact, it exceeded our expectations. When we told the Holo dev team about how many people we hoped to invite, one team member was concerned about pushing it past 300 concurrent users. Well, you all helped us blow past that by more than 3× without any visible problems.

This was exciting, partly because there are a bunch of improvements that didn’t even make it into this test. Sharding was still turned off, none of the recent database work was included, and there are many more optimisations we could explore. Given that sharding is part of our strategy for achieving near-infinite scalability, it looks like performance can only get better — by orders of magnitude.

We also discovered bugs and slowdowns. As Holo wrote in their most recent update, that was the purpose of this test. So even though this might seem like a bad thing, it was really just another marker of success.

Kinda special

When you’ve been immersed in the development of a project for a few years, it’s easy to forget what’s different about it. But as I lay awake in bed a few weeks ago, thinking about our plans to open testing up to the whole world, I started remembering why what we have is special.

Usually, when you want to access a P2P network, you have two options:

  • Become a full participant, which means downloading the software, connecting to peers, and replicating their data. For non-blockchain networks, this is often as easy as downloading an app — as long as your computer’s permissions allow it. (Don’t even try this on a public library computer.) But for blockchain networks, this is nearly impossible — the hardware demands are too high, and installing the software can be a challenge.
  • Access a centralised ‘gateway’, which handles all the P2P stuff for you. If you want to write things to the network, depending on the nature of the gateway, you either need to install a ‘wallet’ browser plugin for your cryptographic keys (most blockchain dApps work this way) or trust the gateway to handle your keys for you (most cryptocurrency exchanges work this way).

I prefer the first option. I get to take charge of my participation in the network, I can work offline if I like, and I get the happy feeling that comes from supporting something that matters to me. While I’ve never tried installing a blockchain node on my machine, I’ve tried other P2P apps, and they’re quite nice. My only frustration is that I can’t easily switch my work to another computer.

The second option lets me work from any machine I like, but I have to be connected to the internet all the time. And if the gateway goes down, good luck getting anything done. To me, this doesn’t seem very decentralised. And besides, I have a really hard time figuring out how crypto wallets work — even after four years in the business. If this is what my experience is like, what’s it going to be like for my dad?

Holochain + Holo hosting can blend the offline-friendliness of self-run nodes with the portability of web access, all while maintaining users’ sovereignty and making their experience pleasant and familiar. I know of very few other projects doing that.

What’s next?

During the test, the dev teams identified some areas of work. You already know about some of them — the last few releases have included bugfixes that came directly from this test. Here are some others:

  • Database optimisations that increase efficiency. One of these, which permitted asynchronous database queries, arrived in Holochain 0.0.115. The other, a large database refactor, was completed in 0.0.117 (see below).
  • Gossip algorithm improvements and other bug fixes to issues discovered during testing, to reduce overhead and make gossip sustainable over time. One issue in particular involves gossip of historical DHT data. When two nodes sync with each other, they compare the data they’re holding using something called a Bloom filter. It’s a small data structure, kind of like a hash, meant to quickly compare approximate differences between two data sets. Currently Holochain calculates it on every round of sync, which results in a lot of database I/O, causing the sync to time out and restart. We believe this is one cause of the slowdowns, and we’re investigating ways to make historical gossip more efficient. Some of this work has already landed in recent releases.
  • And of course sharding. This will likely give us the biggest improvements. Overall efficiency will get better, because fewer nodes to gossip with = less gossip = more time and space to work on other things. ‘Convergence time’ will also improve. In distributed systems speak, convergence is the moment when all nodes agree on the data they’re holding. In Holochain terms, this means that newly published data will become available more quickly, new nodes will sync more quickly with their peers when they first become part of the DHT, and old partitions will heal more quickly.

Now that you know where we’re heading, let’s take a look at what happened with Holochain the past four weeks.

Four new Holochain releases

Holochain 0.0.116: Dependency cleanup

This was a very small release: some unused dependencies were removed, which sped up compilation time. (#1116)

Holochain 0.0.117: Database refactor

The biggest change in this release is that gossip is shared between all cells on a conductor which share a DNA, which allows Holo-hosted cells to gossip more efficiently.

  • Breaking: DHT shard and authored databases are shared among all cells that share a DNA. There’s no breaking changes to the HDK, but your conductor will have to start over with a fresh database. (#1130)
  • Breaking: Wire messages have been changed to support shared gossip, which means that this version’s gossip protocol breaks compatibility with earlier versions. Again, no HDK changes. (#1091)
  • Breaking: The db_sync_level directive in the conductor config has been renamed to db_sync_strategy and can accept the values Fast and Resilient. This directive handles how SQLite writes are flushed to storage. (#1130)
  • Bugfix: Some databases can handle corruption by wiping and recreating the database. (#1039)

Holochain 0.0.118: Small gossip change

This release brings one breaking change to gossip: nodes now keep metrics on each other based on their agent key, not TLS certificate. This once again breaks the gossip protocol’s compatibility with earlier versions of Holochain. (#1114)

Holochain 0.0.119: Performance improvements

This release brings a bunch of small changes that increase performance.

  • Breaking: when syncing the DHT ops they’re holding, two nodes can send the data in batches rather than in one huge message. Again, this breaks gossip protocol compatibility. (#1143)
  • A few cleanup sweeps have been done over the entire codebase, catching database queries that weren’t made async in 0.0.115 (#1145) and various outstanding todos (#1150, #1152).
  • A small database change results in faster get queries. (#1143)
  • Agent info entries are published, not just gossiped. (#1115)

Read the release notes for the last four versions.

Wait, where’s HDK 0.0.119?

You may have noticed that, while Holochain is now at version 0.0.119, the HDK is still at 0.0.115. What happened to the four newest versions?

Based on feedback from hApp developers, the release team has decided to only bump HDK release numbers when there are actual changes. This does cause a mismatch between a HDK version and the Holochain version(s) it works with, but it also means you don’t have to update and rebuild your zomes with every Holochain release.

To find the HDK version that matches the Holochain version you want to use, we’ll be introducing some tools and documents. In the meantime, you can visit this URL (using Holochain 0.0.119 as an example release; you’ll need to change that number to the release you actually want):

https://github.com/holochain/holochain/blob/holochain-0.0.119/crates/hdk/Cargo.toml

In this case, you can see that the HDK version is at 0.0.115:

A screenshot showing the HDK's Cargo.toml file, with its version number highlighted

A first-time hApp creation tutorial

Today I came across this tutorial by wimpheling, a new hApp developer, documenting their recent learnings as a tutorial. This guide takes you through setting up the Holochain dev tools, building a hApp backend, and creating a web-based UI for it. I found it easy to read and follow — and I’m grateful they wrote it, since we don’t currently have a good beginner tutorial that takes you all the way to a running app.

If you haven’t yet built a hApp but you’d like to give it a try, follow the tutorial and see what you think! (Note: it says it’s written for 0.0.117, but it should work for 0.0.119 just fine. One other thing to keep in mind is that, while the blog says to use https://nightly.holochain.love to download the latest dev tools, you should use the non-nightly URL https://holochain.love instead.)

Playground: now with real DHT data!

Video: a couple sequences of Eric Harris-Braun taking a test run of the Holochain Playground

Long-time readers will remember the Holochain Playground, a simulator that helps you understand how all the agents and data in a DHT relate to each other. You could create cells and watch how their source chains and the shared DHT data grew as they made zome calls.

Now, Playground’s creator Guillem has added a ‘connected’ mode, which hooks up to a real conductor and shows you exactly what’s going on in a real DHT! Because I’m not much of a hApp dev myself, I’ll let others describe what they think:

This is amazing, Guillem. Really awesome. [...] Everybody will benefit from using this.
— Eric Harris-Braun, Holochain co-creator
To me, as a hApp developer, this is game-changing. Really next-level in terms of being able to learn as I go and more deeply understand all the mechanics that are going on — and understand bugs and ways to optimise.
— Harlan Wood, hApp developer

Even Guillem discovered fun surprises as they played with a copy of Eric’s Where app.

If you’d like to understand a bit more about why this could be useful to you, watch the Holochain in Action demo. When you’re ready to try it out, download it from the GitHub repo and watch the video again for usage instructions.

Holochain In Action: countersigning

If, like me, you’ve been enjoying the entertaining, explorative, and educational Holochain In Action videos, there are four or five new ones since last Dev Pulse. They cover interesting topics like test-driven design with Sweettest, creating a minimum viable NFT implementation, and more. I thought this one looked particularly educational: countersigning and multi-party signatures. Since countersigning will be an important part of any sort of transactional app, it seems important to get a good grounding in how it works.

You can find all the other videos here.

Cover photo by Moritz Mentges on Unsplash

Install Holochain Learn about Holochain Join the Discussion