I came back from the Holochain & Rust Developer Immersive to find a big pile of interesting things to write about. One, of course, was the Immersive itself, which I’ll share about next week. But for this week, here are the usual weekly release changelogs for the last six releases (there are some important changes in there) and notes from around the ecosystem.

Holochain 0.0.146: Zome IDs re-added to types

HDI compatibility: 0.0.11
HDK compatibility: 0.0.139

As I mentioned in the Dev Pulse that introduced the HDI, zome IDs were removed from entry and link types (and their action headers) but would be reintroduced later. This is the release that reintroduces them, and we’ll have some ergonomics improvements to help you write more concise validation functions in the future. This has resulted in a number of changes (#1453), some of which are very important and others of which are unlikely to be noticed unless you’re bypassing the HDI and HDK and creating your own interfaces to the host.

There are other unrelated changes:

  • Breaking/New: x_salsa_poly1305_* functions have been properly implemented! As a consequence, any KeyRefs previously stored in the Lair keystore will no longer work. This means that the conductor now uses Lair keystore 0.x.x (currently 0.2.0 at time of this release). (#1446)
  • Breaking/New: In preparation for rate limiting, the action structs that support app-defined weights now have a weight field. While the conductor doesn’t currently use this field, in the future you’ll be able to use it to write deterministic validation functions that turn DHT spam into a warrantable offense.

Read the full changelog.

Holochain 0.0.147: Quantised Gossip

HDI compatibility: 0.0.11, 0.0.12
HDK compatibility: 0.0.139, 0.0.140
Network protocol compatibility: Holochain 0.0.147

This Holochain release was a quiet rollout of a change we’ve been working on and getting excited about for a while: a new syncing strategy for rrDHT called Quantised Gossip. It promises to help DHTs reach consistency faster, with less network traffic. We think it’s an innovation in DHT design, so much so that we filed a patent application for it.

If you’re not familiar with how rrDHT works, it’s fairly straightforward. People and data both get an address that corresponds to their identity — public keys for people, hashes for data and metadata. These addresses live in the same space, a ring of numbers that starts at 0 and wraps around to meet its start at 2^32, or about 4.3 billion.

Each person claims responsibility for storing data and knowledge of their neighbours for a portion of that ring. When those ‘arcs’ of responsibility overlap (and they should), they create a ‘neighbourhood’ that keeps data resilient against loss. Holochain tries to maintain a lot of overlap, so that participants can come and go with reliable data availability.

But the traffic of people entering and leaving the network requires the existing peers to adjust their coverage, and new data is being created all the time. You can never be quite certain what your peers already know about and how big of an arc they're holding at any given moment, and it’d be costly to simply push everything you think they needed every minute or so.

So, in rrDHT, peers compare the list of data they’re each holding in their overlap and only send each other data when something has changed. You can read more about this in the intro to rrDHT, the Core Concepts article on the DHT, and this small Dev Pulse section about a recent change to arcs.

Previously, we were using something called a Bloom filter. It’s like a compressed list of hashes, and it takes less data on the wire. But it turns out that it’s costly to calculate. And you can’t precalculate it; remember that you and your neighbours are regularly adjusting your arcs of responsibility to ensure that no portion of the ring is under- or over-replicated, thus changing the region of addresses in the overlap.

The new Quantised Gossip comparison algorithm speeds up rrDHT by ‘chunking’ up the DHT into a grid of regions. You see, data in a Holochain DHT could be said to exist in a two-dimensional spacetime — where space is the address of the data and time is, well, the time it was created. This lets peers line up their arcs of responsibility against a coarse grid and create a ‘fingerprint’ of the data inside each region. Then, all they need to do is compare the fingerprints of all the regions they have in common, and only share data for regions whose hashes differ.

Why is this faster? First, you can precompute the fingerprint for each region you’re storing and recompute it cheaply every time you receive new data. That means you’re ready for the next time you gossip with your neighbours, even if you or they have changed your arc of responsibility slightly. It’s easier to share ten precomputed fingerprints than to recompute a fingerprint for an arbitrary arc that’s 65,000 addresses wide every time you chat with a neighbour.

Second, when you chunk data into time-based regions, you can save bandwidth by making the older time regions larger and fewer, because they’re less likely to change. The dance with Quantised Gossip is to balance spending bandwidth on comparing regions versus actually transferring DHT data. Regions that are less likely to see updates can be larger, because the risk of them changing is lower.

[Ed: This paragraph has had some technical errors corrected since publication.] rrDHT already had two schedules for gossiping — ‘recent gossip’ for everything that's been created in the past 15 minutes, and‘historical gossip’ for everything else. This new algorithm replaces Bloom filters with Quantised Gossip for the historical gossip, making that process much more efficient. Even though we’re not yet doing optimisations like precomputing region fingerprints, we’re already seeing reductions in network traffic.

There are a lot of subtle details in here — for instance, peers estimate the data density in a region and the probability of discrepancies with their neighbours in order to balance region comparison bandwidth versus data transfer bandwidth. There’s plenty of room for future optimisations, and the general mechanism is built and ready to receive those optimisations.

Read the fantastic documentation on Quantised Gossip and the full changelog (note: doesn’t contain any notes, just links to the list of versioned crates in the release).

Holochain 0.0.148: Enzymatic countersigning functional

HDI compatibility: 0.0.11, 0.0.12
HDK compatibility: 0.0.139, 0.0.140
Network protocol compatibility: Holochain 0.0.148 and above

Holochain 0.0.145 introduced changes to ‘enzymatic countersigning’ (explained below), including the ability to add optional signers. At that point it wasn’t yet operational — the required data structures were simply ready for future functionality — but now the networking layer fully supports it. (#1472)

So what are enzymatic countersigning and optional signers, and what are they used for? When you enter into a countersigning transaction with another party, normally you all send your countersigned entry along with your signatures to the DHT peers who claim responsibility for the entry’s hash, then wait for them to collect all your counterparties’ signatures and send them back to each one of you. In this scenario, the DHT peers are ‘driving’ the session — taking responsibility to help all the counterparties reach consistency and commit a transaction to their chains.

But because a distributed network isn’t always consistent, it’s possible for one party to think the transaction failed. This can happen because the DHT peers they’re communicating with failed to reach consistency in time, or they were slow in sending back the full signature list. Or maybe they all happen to be mailicious. The counterparty might then back out of the session and write another entry to their chain. This puts them in an inconsistent state, because to everyone else it looks like they agreed to the transaction but later ‘forked’ their chain.

Enzymatic countersigning partially solves this problem. In this scenario, one of the counterparties (who could be a transacting party or simply a third-party witness) drives the session, collecting all signatures and sending them back to the other counterparties. As long as they’re honest and have good uptime, it’s a lot less likely that one counterparty will end up in an inconsistent state. The enzyme could still go offline or take time sending the message to one counterparty, but there’s never any inconsistency from the enzyme’s point of view. The other counterparties can double-check with the enzyme before they make their next move.

Optional signers lets you add a certain number of additional witnesses to a transaction. This is useful if you need a quorum of signers from a list of candidates but you don’t care which of them actually do the signing. Some signers can be offline and the session will still complete — as long as the enzyme is online. That’s because an enzyme is a necessary participant when optional signers are involved; they’re the one who determines exactly which optional signers get recorded as counterparties, so that everyone can validate the transaction in a consistent manner.

Read the full changelog.

Holochain 0.0.149: Documentation

HDI compatibility: 0.0.11, 0.0.12, 0.0.13
HDK compatibility: 0.0.139, 0.0.140, 0.0.141

This release added top-level crate documentation for the HDI library holochain_deterministic_integrity and a section on integrity and coordinator zomes in the HDK documentation.

Read the full changelog.

Holochain 0.0.150: holochain_deterministic_integrity → hdi

HDI compatibility: 0.0.11, 0.0.12, 0.0.13, 0.0.14
HDK compatibility: 0.0.139, 0.0.140, 0.0.141, 0.0.142

Breaking: The holochain_deterministic_integrity crate’s name has been shortened to hdi. There are no internal changes in this crate besides the rename, a version number bump, and more documentation updates.

Read the full changelog.

Holochain 0.0.151: Network seed, deterministic DNA bundling, Validation Ergonomics

HDI compatibility: 0.0.11, 0.0.12, 0.0.13, 0.0.14, 0.0.15
HDK compatibility: 0.0.139, 0.0.140, 0.0.141, 0.0.142, 0.0.143

There’s a small collection of important changes in this release:

  • Breaking: The uid property in DNA manifests, which changes the DNA hash and ‘forks’ it into a separate network space without changing any of the DNA’s functionality, has been named to network_seed. This better reflects its purpose. (#1493)
  • New: The genesis_self_check callback now has access to dna_info() and zome_info(). As this callback is an opportunity to ‘prevalidate’ a membrane proof before trying to join a network, it’s crucial for it to have access to things like the DNA’s properties block which may have information necessary for membrane proof validation. (#1491)
  • Breaking/New: The payloads of the Op enum variants have changed; now they are all tuples that contain a single struct. There are also some new op helpers available in the HDI to make validation callbacks nicer to write — specifically, it gives access to the most common bits of an operation as well as your defined entry and link types, making it faster to get to the thing you actually want to validate. Check out the documentation for OpHelper and OpType, as well as the tests, to see how you can structure your match arms for ops. (#1488)
  • Bugfix: DNAs are now bundled deterministically, which means that a collection of zomes and a DNA manifest file will now always result in the same DNA hash when hc dna pack is run. (#1479)
  • As always, there are more documentation updates for the hdi and hdk crates. Dear reader, I know that I mention this in every release announcement and you may be getting bored, but the developer experience advocate in me gets a little thrill every time I see that a new component gets more thorough documentation.

Read the full changelog.

Some notes on how we name and number our SDKs

Since we’ve changed the name of the HDI crate, I thought I’d review what the two core SDKs for writing hApps are about.

  • Holochain Deterministic Integrity (HDI) is an SDK that lets integrity zomes interface with the conductor that hosts them. Integrity zomes are the packages of code that define rules of valid behaviour in an application’s network, including the shape of public DHT data. It only gives code access to purely deterministic functions — the minimum required to get the job of validation done.
  • Holochain Development Kit (HDK) is a more full-featured SDK that gives coordinator zomes access to everything in HDI plus functions that have side effects, such as writing to a source chain or pinging another participant, and non-deterministic functions, such as generating a random number or querying the DHT for a collection of links.

I’ve always been a bit uncomfortable with the name HDK, because there’s already a perfectly good name for that sort of thing: an SDK (and besides, ‘HDK’ already means ‘hardware development kit’). So while the HDK’s crate will still be named hdk, we’re going to call it the Holochain SDK whenever we’re speaking about it to general audiences. That should make it clearer to new developers what it’s about and what it’s used for.

Additionally, some people have pointed out that we promised that HDI 0.1, the ‘Stable Validation’ milestone, was going to be ready in time for the Developer Immersive, and we’re still only at version 0.0.14. It is indeed true that we didn’t bump the version number to 0.1, but the core devs did meet their target: a stable, feature-complete, relatively frozen HDI ready for course participants to start hacking on and using long-term.

We still have some more changes to make (partly based on feedback from course participants), and then we’ll likely cut a 0.1 release. In the meantime, the HDI’s API isn’t likely to receive any breaking changes — only updates.

electron-holochain and holochain-runner updates: Holochain 0.0.149, Lair 0.2

electron-holochain and holochain-runner, two projects from dev shop Sprillow that bundle up the conductor, Lair keystore, and your hApp into a single binary, have received updates that bump the supported Holochain version to 0.0.149 and Lair 0.2.0. If you’re using either of these projects in your development workflow, you can now update to newer Holochain support.

Holochain Launcher: Two releases

0.4.9: Get your hApp into the hApp store!

This release updates the bundled version of the DevHub hApp, the “distributed app for distributing distributed apps”, to 0.7.1. In addition to various data model changes, this version sets the foundation for reviews, reactions, and ratings. And the Launcher now recognises when you tag your hApp release with the app-store-ready tag and will display it in its built-in App Library. The App Library already made it as easy to install a hApp as it is to install an app on your smartphone; now you can easily publish your production-ready hApps too!

Note: There are no new conductor, Lair, or HDK releases supported in this version.

0.4.10: Bump Holochain support to 0.0.150

This release adds support for Holochain 0.0.150 / HDK 0.0.142, and removes support for everything older than Holochain 0.0.143 / HDK 0.0.136.

Q&A with hREA project

This is a project I’m very excited about. It’s hard to explain exactly what it is, so I’ll just call it a toolkit for building applications that support cooperative economies that are rich with information beyond mere price. I think it has the capacity to redefine our view of the economic world, which could also shape our relationship with the world outside of economics and lead us into a more respectful, regenerative presence in this planet. (If you doubt that accounting systems can have an impact on the world, consider how double-entry bookkeeping, with its isolated entities receiving inputs and producing outputs — its rejection of every uninteresting effect as ‘externalities’ — has caused us to overshoot our planet’s capacity to sustain us.)

hREA is gearing up for a major milestone release, likely sometime this fall, so they’re deep in development. Their team is expanding and I’m seeing design and engineering discussions flying around left and right.

The hREA team recently invited the world to join them in a Q&A session about the project. If you’re interested in using software for building sustainable marketplaces and economies, check out this video to learn what they’re about:

But what about the Developer Immersive?

It was amazing. Powerful instruction, rapid learning, wonderful people, beautiful scenery, good food. I’ll share a lot more next week. Until then, here are a few photos!

Students work together on exercises at the Developer Immersive
Developer Immersive participants visiting a waterfall on the weekend
Instructor Guillem explains some CRUD functions of the HDK in the lecture hall
Students and staff at the Holochain & Rust Developer Immersive, July 2022

Cover photo by Vitolda Klein on Unsplash

Install Holochain Learn about Holochain Join the Discussion