Big HDK Change

This week’s Holochain release brings a substantial change to the HDK: validation is based on DHT ops, not source chain elements. This change is coming now as part of the core dev team’s effort to stabilise the HDK so you don’t have to refactor your code as often.

There’s a lot going on in the hApp ecosystem as well. Read on for all the news!

A note about breaking HDK changes

Before I share the release notes for the last two Holochain versions, I want to talk about what I mean when I describe some HDK changes as ‘breaking’. Generally I mean one of two things:

• Something in the hApp-dev-facing ‘surface area’ has changed — that is, public functions or types — and you need to refactor your code.
• The hApp-dev-facing surface area hasn’t changed, but something in the conductor’s host API has changed behind the scenes, which requires you to recompile your zome code against the new HDK.

But this morning the core devs educated me about another, more subtle breaking change. This change happens every time you update to a new HDK version, even if there aren’t any of the above sorts of changes. In fact, it happens every time you recompile your zome code — even if you stick to the same HDK version.

What is this breaking change? Every time you recompile, the DNA hash changes. This is because the Rust-to-WASM compiler doesn’t produce deterministic builds — at least, not without a lot of tooling work. We don’t have a step-by-step guide for setting up deterministic builds, and from what I understand it’s quite a difficult process.

I’m still searching for better language, to distinguish different kinds of break. Until I find the right words, I’ll just clarify whether a change breaks your code versus compatibility with the host API. If I say ‘no breaking changes’, however, know that you can still break compatibility with prior copies of your DNA’s compiled WASM simply by recompiling — HDK upgrade or not.

Holochain 0.0.126: Bug fixes, HDK consistency

Most of the work in this version was administrative — CI improvements, test coverage, and the like — there are three things worth knowing about.

• The HDK has been updated to 0.0.122, with no breaking changes.
• A panic on conductor startup has been fixed. (#1206)
• The HDK’s entry deletion functions delete, delete_entry, and delete_cap_grant now take a new DeleteInput struct, which lets you specify relaxed chain top ordering just like create and update functions. This is not a breaking change, as you can still pass just a plain HeaderHash as before and it will get converted to a DeleteInput. (#1213)

Read the full changelog.

Holochain 0.0.127: New validation approach, other HDK changes, gossip changes

The biggest developer-facing change in this release is a new approach to validation based on DHT operations rather than source chain elements (#1211, #1212, #1210). And it’s a big breaking change. Here’s what you need to know:

• When an agent publishes a source chain element, it gets converted into a number of DHT operations that transform the data that other peers store. There are three kinds of operations: ones that create an entry, ones that create a header, and ones that affect metadata on an entry, header, or agent ID (such as a link, delete, update, or source chain change). You can get an overview of this in the core concepts.
• Moving validation to operations forces you to think more deliberately about the previous point. This may make things complicated at first, but I feel that it also brings clarity and helps you reason more accurately about what’s happening on the DHT when an agent publishes one of their source chain elements.
• Each zome has only one validation callback which receives all operations it produced, as well as all operations not produced by any specific zome (such as those generated from system elements like the membrane proof or agent ID elements). All other validation callbacks are now obsolete.
• The validation callback receives one parameter, an enum called Op, that describes all seven kinds of operations and contains the appropriate element and entry data. The documentation describes the enum in depth, telling you which operations are created from which types of source chain elements.
• If you look at the documentation for Op, you’ll notice that agent activity ops (changes to an agent’s source chain that get registered as DHT metadata on their agent ID entry) can now be validated.
• The validation package, which included a portion of the author’s source chain, is no longer passed to the validation callback. This input field had been empty for a while and will eventually be replaced by a host function which lets you query portions of the author’s source chain via the DHT. (For now you can iterate over the source chain using must_get_header repeatedly.) All other dependencies can be retrieved using must_get_* functions, as usual.
• You can see code examples for the new validation approach in the wasm_workspace tests folder (scroll down to ‘validate’).

Here are the other changes in this release:

• Sharding is now turned on by default (#1237). That means that each participant has less work to do in helping secure the integrity of an application’s network and data.
• The HDK has been updated to 0.0.123, with a few breaking changes in the HDK.
• The init guest callback, called by the conductor when it runs a DNA for the first time, can now use the call host function, allowing one agent to access functionality in other zomes within the same cell, other cells with different DNAs on their own machine, or other agents’ cells with the same DNA. (#1186)
• The host exposes a new hashing utility function called hash with a lot more options. It’s not exposed via the HDK, but it does show up as a number of convenience functions, including a refactor of the existing hash_entry. Now you can pass anything hashable — not just entries, but also headers and even raw bytes. If you’re passing raw bytes, you can receive either a 256-bit Blake2B hash (the default used by Holochain), a 256-bit Keccak hash (used by Ethereum), or a 256-bit SHA3 hash. (#1222, #1228, #1230). Support for 256- and 512-byte SHA2 hashes is also planned.
• Breaking: Your DNA manifest now takes an origin_time field, which defines the earliest timestamp that any source chain header may have. This prevents agents from forging nonsensical timestamps, but will also help with the efficiency of the not-yet-released fast gossip algorithm. Timestamps can be specified as an RFC3339 time string, or the number of microseconds since the start of the UNIX epoch if you’re good at math. If you don’t specify an origin time in your manifest, it’ll default to January 1st, 2022 at 00:00:00 UTC (maybe we should call this the start of the Holochain epoch!) Unspecified origin times will eventually be deprecated. When you use the hc dna init command, it’ll populate the manifest with the system’s current time as an RFC3339 time string. (#1224)
• Breaking: The P2P protocol has two changes which break compatibility with version 0.0.126 and lower. First, hashes are no longer passed along with data, as authorities need to calculate these anyway. (#1233) Second, in preparation for the new fast gossip algorithm, nodes can sync DHT data by ‘region’ (more on this when the new algorithm lands). (#1234)
• There are also lots of updates to documentation courtesy of Jost, as part of our increased commitment to improving all developer documentation.

Read the full changelog.

DWeb webinar on decentralised storage

Our friends at the Internet Archive and DWeb are hosting monthly webinars on various decentralised web topics. The next one, on Thursday, Feb 24th (tomorrow if you’re reading this fresh after publishing), is on decentralised storage. Read more details and sign up here:

Decentralised storage on Holochain with the file-storage module

On the topic of decentralised storage, I must mention the file-storage module created by Guillem Córdoba, Hedayat Abedijoo, and Harlan Wood. It accepts any kind of binary data, chunked into pieces 16MB or less for more equitable distribution around the DHT, and identified by a file manifest that lets you name it, give it a type, and specify who created it and when.

It includes examples of two DNAs: one in which everyone can write and retrieve data directly to and from the DHT, and one where they do it through peers who have agreed to be ‘providers’. This second approach could be used to create hosting incentives for certain files, for example.

Chunking is the responsibility of whatever client you create, because different chunking strategies might make more sense for different contexts. You may want to align your chunk boundaries along video frames, or optimise for lag-free streaming or deduplication, or play with chunk sizes that have an impact on the amount of data each peer is asked to store. There is an example UI included that breaks files into 256 KB chunks before committing them to the DHT.

All in all, it’s pretty comprehensive! This module has actually been around for a long time, and I may or may not have mentioned it in a past Dev Pulse, but this seemed like a good time to bring it up!

Video: Scaffolding a hApp in less than 10 minutes

This is a workshop that Guillem Córdoba in the Web3 Infrastructure room at FOSDEM 2022. It starts with a great introduction to what Holochain is about, then proceeds to show you how to get started writing a hApp using our scaffolding tool, all while explaining how a hApp is architected. I haven’t tried it myself, but the tool looks both easy and comprehensive — it even lets you generate a basic UI!

Video and demo repo: Bundling your hApp with Electron

In the most recent Holochain in Action video, Connor walks you through the process of bundling the Holochain conductor, your hApp, and your web-based UI into an Electron-based executable, using his template. If this is something that you find exciting, you’ll probably want to look at this example repo that not only uses the Electron template but demonstrates how to use GitHub CI to do the building for you.

Forum conversation: Migrating data on DNA upgrade

The question of migration — what to do with all the data in an existing DHT when it’s time to upgrade the DNA — is a good one. There are design challenges involved, so it’s no surprise when someone brings up questions about it.

This forum thread about migration is a good one. Connor Turland of Sprillow lays out some good principles for migrating a hApp’s data and hints at the possibility of future changes that will make migration a less common occurrence. (Note: these changes are very likely to happen, but the details are still being worked out.)

Correction: dev.hc Discord invite URL

In the last Dev Pulse I made the mistake of sharing an invitation to the dev.hc Discord server that eventually expired. Here’s a permanent invite URL; we’d love to see you there!

The dev.hc Discord server is created and stewarded by the community; it’s a great place to have informal conversations about developing Holochain apps or understanding the technical details better. And don’t forget the Holochain Forum — not only is it a good place to have more in-depth conversations involving a lot of text and code, but there are a lot of great resources in past forum threads!

Cover photo by Annie Gray on Unsplash