Last week’s Holochain release brings improvements on many performance fronts. It tackles some database query inefficiencies, bringing some welcome reductions in RAM and CPU usage. Meanwhile, the developer community is exploring last the previous release’s breaking changes to validation, and have produced some materials worth watching if you’re trying to understand what these changes mean for your hApp.

A second note about breaking changes

This is a follow-up to my previous note. From now on, I’m going to be more explicit about breaking changes for each release, by starting the release notes with a big bold warning about anything that’s broken compatibility in the following seven categories. By ‘breaking’ I only mean something that is incompatible with the previous release of Holochain or any other component I’m reporting on — not the change in DNA hash that happens whenever you recompile a zome as described in last Dev Pulse.

• Network protocol: This is the gossip protocol that Holochain peers use to talk to each other. No updates are required for your zome code, except when a network message changes the data passed through the host API.
• Host API: This is both the API that the conductor exposes to your zome code, such as source chain write functions, DHT retrieval functions, and crypto functions; and the API of callbacks that the conductor expects your zome code to implement, such as init and validation functions, and any arbitrary functions that make up your zome’s public API. A breaking change in the host API always results in a breaking change in the HDK and will require you to update your zome code to match.
• HDK: This is the Rust-based SDK that your zome code can use to interact with the conductor. If the host API introduces breaking changes, the HDK will always change. Occasionally it will change in a way that’s invisible to you as an application developer, but even here your zome code will need to be recompiled in order to work with the updated host API. And occasionally the HDK can change in ways that maintain compatibility with the host API, such as December’s changes to Path.
• Database: This is the SQLite database that holds users’ source chain data, the DHT shard they hold, and DHT routing and health information. If its schema changes, any application data stored by previous releases of the Holochain conductor will no longer work with the new release. No updates are required for your zome code, except when a change is caused by a change to information passed through the host API.
• Conductor APIs: The conductor exposes these APIs to any client running on the user’s machine via a local HTTP connection that sends and receives MessagePack data as WebSocket messages. There are two APIs: the admin API, which lets a client manipulate hApps, agent keys, and change conductor configuration; and the application API, which lets a client actually call a hApp’s exposed zome functions. Any change in these APIs will also change the JavaScript and Rust conductor client libraries.
• Conductor config format: User and developer machines specify conductor configuration in this format. Changes to it will require users and developers to modify existing config files, as well as scripts that specify conductor config as command-line arguments.
• Keystore protocol: The Lair keystore, which stores all the user’s cryptographic secrets on their machine, has a protocol that it exposes to their conductor over a local socket. If this protocol changes, the conductor will come bundled with a new Lair binary, and the new keystore file format will sometimes also change, requiring users to create a new one.
• Bootstrap API and proxy protocol: The bootstrap and proxy are two semi-centralised services that help peers discover each other and communicate through firewalls. The conductor talks directly to them using their APIs/protocols. If those change, both the conductor and the bootstrap and proxy binaries (bundled in the Holonix development environment) will change, and our publicly hosted bootstrap and proxy instances will be upgraded too.

Holochain 0.0.128: Database query improvements

Breaking changes warning: network protocol, bootstrap API

In preparation for Holo’s ‘long-term app testing’ milestone, the core dev team wrote a performance test that would simulate a DHT with 15 GB of chat data, hosted by ten conductors — essentially trying to recreate how Elemental Chat would be performing by the end of our multi-week testing run. Initially, the test result wasn’t that great: it couldn’t even complete on a fairly high-powered development machine, as CPU consumption went to 100%.

Now, after a lot of work (#1243, #1244, #1245, #1246, #1247, #1248) that same test runs on the same development machine with each conductor using only 2% CPU and 65 MB of RAM, for a total of 20% CPU and 650 MB RAM. That’s an impressive improvement!

Now, it’s important to make meaningful comparisons. This doesn’t mean that every happ will run like this on any sort of computer — some will consume more resources; others less. But this test shows that, even under high gossip load, Holochain is now doing much better than it was a few weeks ago — and it’s more than capable of supporting a public and heavily-used Elemental Chat application.

It also makes Holochain more viable for low-powered systems such as Raspberry Pis and smartphones. This has always been one of our guiding principles. This helps companies like RedGrid achieve their goal of running Holochain on IoT devices.

Note: on 3 March RedGrid rejoined our commercial director David Atkinson in a live-streamed interview to share some status updates on the Internet of Energy Network. I found the whole thing really inspiring; I hope you do too!

(Note: storage space has increased somewhat, because one of the fixes introduces new database indices.)

Other changes in this release:

• The old simple-bloom gossip algorithm, which is no longer used, was removed. (#1256)
• Breaking (bootstrap API): The conductor can now get a list of available proxies from the bootstrap server. This feature can be configured in the conductor config. If you’re one of the rare folks running your own bootstrap server, this breaks compatibility with the old bootstrap server and you’ll need to update to kitsune_p2p_bootstrap version 0.0.4 in order to keep serving your users. (#1242, #1236)
• The entire Holochain codebase has been updated to Rust 2021 edition. (#1253)
• Breaking (network protocol): Gossiped DHT operations no longer include hashes, because the receiver needs to calculate them anyway during validation. This saves 39 bytes of wire data for each operation. This breaks compatibility with the network protocol of Holochain 0.0.127 and earlier. (#1250)

Read the full changelog.

Holochain 0.0.129: Codebase cleanups

Breaking changes warning: none

This release has no breaking changes for anything concerning developers or users. It does, however, contain a lot of work dedicated to cleaning up the codebase. To a non-developer, this might seem like pointless work. But even though you don’t ever see what the kitchen looks like in your favourite restaurant, you kinda do care about that, don’t you? You hope the floors are clean and there are no rats running around.

So it’s important for the restaurant crew to keep a clean, orderly kitchen. Mess happens all the time; it’s just part of running a busy kitchen. The same is true of a codebase; you need to do regular checkups to clean up and make sure everything’s in the right place and in working order. This makes it easier for all the core developers to work, and in this case also improves performance in a couple places.

Here’s a sampling of what they’ve been working on in the kitchen:

• Rename database code to remove assumptions based on our previous database engine, LMDB. (#1270)
• Demonstrate a different approach to interfacing the conductor and the networking library. (#1260, #1267)
• Remove unneeded test structures. (#1271)
• Improve reliability of automated test and build (continuous integration) workflows, which are run by GitHub every time a release is cut or code is being considered for a future release. (#1273, #1274, #1275, #1276)
• Add a config param to create key log files for debugging TLS sessions. While this changes the config file format, it’s an additive change, not a breaking one. (#1261)

Known issues

• Unmet dependency shuts down app: Rather than failing validation on an unmet dependency and retrying after fetching it, the conductor treats this as an unrecoverable error and shuts down the application. This may also be true for other types of application errors. This is already being worked on (#1279).

Read the full changelog.

Lots of documentation work

I may have mentioned this already, but even if I have it’s worth mentioning again. Our new colleague Jost is working to improve the consistency, thoroughness, accuracy, and accessibility of the documentation. Currently he’s working in four different areas:

• HDK reference
• Conductor API references (admin and application APIs)
• Conductor config reference
• Installation guide

In particular, the installation guide is getting some updates that match more closely how developers actually use Holochain; this should help new developers get up and running with a setup that’s been tested and refined by others. In the future the release team will integrate new tools that make updating Holochain and your hApps to a new version easier.

Holochain Launcher 0.3.10

The newest version of this end-user Holochain conductor and hApp installer has bumped the internal version of Holochain to 0.0.127. You can get it from the above GitHub link.

Updates to holochain-runner and electron-holochain-template

These two projects from Connor Turland at Sprillow help you build a self-contained app with all Holochain components bundled into one binary. holochain-runner wraps the Holochain conductor and keystore into a self-contained binary that runs a single hApp, while electron-holochain-template helps you build an Electron-based hApp using holochain-runner.

holochain-runner has seen a new release, 0.0.35, which bumps the included conductor to 0.0.126 (compatible with HDK 0.0.122). You can get it on GitHub or crates.io. electron-holochain-template’s main branch has also been updated to match.

Exploring new validation approach

As I reported in the last Dev Pulse, Holochain 0.0.127 introduced a large breaking change in the way validation works. In summary, each zome now handles validation in a single validate() callback that takes one DHT operation, matches on its type, and performs validation appropriate to the operation type and the data the operation contains.

The developer community has been exploring what this change means for their apps. On that theme, the most recent Holochain In Action session is all about validation…

And Eric Harris-Braun has shared a code example from a library he’s working on (more on that in the next section).

Next up, as part of the aforementioned documentation effort, the HDK doc for validation is getting some love, so you can understand exactly how it works and what you should validate for each operation type. Here’s the most recent edition; give it a read and share your feedback with us.

New library: attestations

This library by Eric Harris-Braun gives your hApp the power to create attestations. What is an attestation, you ask? It’s fairly simple: someone claiming something about a subject. Humans do this all the time, so it’s pretty easy to come up with examples: Alice claims that Bob is her son, the driver’s licensing office issues a card that says Carol is allowed to drive, Dave’s university professor Eve signs a certificate that shows he completed his master’s degree. You can see some exciting use cases for this library involving DAO/Discord/NFT mashups, Holochain membrane proofs (the ‘passport’ into a hApp’s DHT), and reviews/ratings/badges/certificates.

With public-key cryptography, you can create a ‘portable’ or ‘self-verifying’ attestation — a data structure that carries a signature that proves that the proper person made the attestation. All you need is the public key of the person who did the signing (as well as a way of knowing that the public key actually belongs to the person you think it does — which is a whole other story).

This library’s readme points out that, in a sense, every source chain element is already a cryptographically verifiable attestation. When it gets published, it carries a public key and a signature that together let anyone else verify the identity of its creator.

The library is pretty simple — basically just enough structure to let anyone in a DHT make an attestation about someone else, plus functionality to search for attestations making a given claim or being made by or about a given agent. I understand there’s also plans to add support for W3C Verifiable Credentials.

New library: threaded-comments

I discovered a small but useful library called threaded-comments. It does exactly what it sounds like it does, adding support for comment threads to your hApp. It comes with a basic but working demo UI so you can play around with it. It doesn’t make assumptions about what sorts of things can have comments attached to them; it just takes an entry hash and a message as input (in fact, that’s what gives you threading for free — you can attach a comment to a comment). This means you can add comment support for anything your app deals in: pictures, blog posts, tweets, transactions, game moves, etc.

This library is coming out of the Wednesday hackalong workshops, in which members of the DEV.HC Discord get together to teach/learn Holochain by working on actual projects. The next workshop (which happened the day this Dev Pulse was published) focused on a reactions library; I hope to share its Git repo in the next Dev Pulse.

It’s really exciting to see more and more developers getting interested in Holochain, gaining experience, and perhaps more importantly supporting each other through the journey. I think of Holochain as a ‘tool for conviviality’, in the words of philosopher Ivan Illich — a tool that gives regular folks the power to take care of each other and build society without excessive intervention by institutions. And it’s so nice to see the developer community practicing the same conviviality they aim to foster with the tools they’re building.

Cover photo by Mitchell Orr on Unsplash