As I pored over the most recent dev updates, I struggled to find a cohesive story to share. There’s no one thing I can point to and say “See, this is what’s happened since the last Dev Pulse.” Just a lot of small updates. And then I realised — that is the message. Right now we’re making consistent, steady progress. Components are being optimised and rewritten, bugs are being found, core apps are seeing accelerated development. For those who would rather not read a tweet stream, here are the major points:

• HPOS, the HoloPort operating system, has seen a version bump of Holochain that’ll make integration of all the upcoming improvements a lot faster. This has involved a bit of work updating the Holo hosting components — Chaperone/Envoy (the parts that connect the user to their chosen host) and the HoloPort/Holochain API.
• One of the upcoming improvements is a new proxy server, which is substantially faster than the previous one. The devs are currently working through a bug in which gossip is sometimes slower than before, and then it'll go to testing.
• Another improvement is a new database layer built on SQLite. It's is unoptimised and currently slower than the current one built on LMDB; our expectation is that it will enable optimisations that make it as fast or faster, as well as new Holochain features (such as sharding) that make the entire network more efficient.
• There are now low-level components that install hosted hApps onto HoloPorts (previously the hApps were hard-coded).
• DeepKey, our distributed public key infrastructure application, is in active development. It features the most complex validation code of any hApp we know of, and requires rigorous testing. So we’ve introduced a mocking framework that lets you unit-test any of your functions that use the host API.
• Elemental Chat’s registration process is getting support for joining codes, so we can open it up to the wider world of the web while maintaining a manageable number of alpha testers. This feature will be available to any Holo-hosted hApp that needs ‘membrane proofs’.
• Rigorous testing of the Elemental Chat registration process has revealed a few bugs in the way Holochain handles cell creation.

That last bullet is an interesting one, and I think it might help you understand just a bit more about what’s going on under the hood if I try to explain where those bugs are coming from.

The bugs we’re seeing are all about setting up a ‘cell’ for a new web user on their assigned HoloPort. A cell is a combination of an app binary (or DNA) plus an agent ID, and every agent who wants to use the app must have one running. (In fact, if there are no cells — no users actively using the app — the app can’t really be said to exist.) The cell is how you (or rather, the UI on your computer) interact with your own data in your source chain and others’ data from the DHT.

Preserving your agency

As you probably know, Holo hosting is a compromise between the full P2P, agent-centric world of Holochain and the centralised world of the web. But one thing we refused to compromise was your agency, which is most powerfully expressed in your ability to choose what goes into your source chain. That source chain is a public record of the things you’ve done in the app, and you don’t want anyone else messing around with it.

In order to preserve agency at this level, a HoloPort should never be able to write to your source chain, even if it wanted to. You see, every source chain entry has to be signed by your private key, which is a secret component of your agent ID. The signature has to cryptographically check out against your public key (which appeared in element #3 of your source chain). And that private key is generated and safely stored in your browser every time you log in. That’s what protects you from being impersonated by a rogue HoloPort.

The way Holochain apps work, when you want to perform an action (say, publish a tweet) your UI calls a function in the cell, which ‘stages’ a few writes to the source chain. Then the Holochain runtime validates those writes against the DNA’s validation rules, creates a signature for each of them using your private key, and writes them to storage.

But this creates a bit of a problem if your private key is on your machine and your cell is on a HoloPort. So the HoloPort has to send a signing request back to your browser, consisting of all the source chain writes it wants you to sign. This happens in a separate HTTP cycle.

Now there’s another problem: there’s a very thin window of time, between making a function call and signing the writes, where you might just close your browser and scuttle your HoloPort’s attempt to get those writes approved. If that happens during registration, where your HoloPort is trying to write your Elemental Chat invite code and public key to your source chain (the genesis elements), it can’t actually create a valid cell for you.

So the Holo dev team is working on addressing that edge case. There are two parts to this: one is to fix an issue where the Holochain conductor locks up on failed validation (that’s just a common bug with an engineering solution on the way), and the other is to recover gracefully when the genesis elements can’t get signed on first try.

Keeping bad actors out of the network

With Holochain, every application (or rather, every DNA within every hApp bundle) enjoys its own isolated network with its own separate store of data (its DHT). And sometimes it makes sense to restrict access to that network.

For this we have the ‘membrane proof’, an element near the beginning of your source chain, that supplies some sort of evidence that you should belong to the network you’re trying to join. Depending on the app, it could be a simple invite code, or it could be something complex such as a cryptographic certificate signed by three existing members. This membrane proof is something you supply when you start up your cell, and it’s validated by your initial bootstrap peers before they let you access the rest of the network and its data.

Elemental Chat is one of those applications; it requires an invite code on signup. The Holo single-sign-on dialog has a field you can paste your invite code into; this gets passed on to your assigned HoloPort.

But what happens when you get the invite code wrong? The validator network should reject you before you even get the chance to take a look at the data in the DHT. This is tricky, because you’re supposed to self-validate before joining — that prevents you from publishing bad data. But if validating the invite code requires access to DHT data, you might not be able to self-validate!

This chicken-and-egg problem has sparked a series of discussions about the best way to handle membrane proof validation. It’s an example of the sorts of engineering challenges the dev team face regularly, challenges in which they have to consider the impact on user experience while maintaining commitment to a ‘correct’ design that ensures data integrity. (I love observing their conversations; it makes me feel smart but also gives me a sobering appreciation for how much I still have to learn about Holochain’s inner workings.)

The first step is to fix a bug that locks up Holochain when the Elemental Chat invite code is invalid. As with the previous section, this is just a routine engineering matter. The more important step, currently being worked on in parallel to the bugfix, is to identify the most sensible design for membrane proof validation — one that is easy for developers to understand and use, ensures correctness, and minimises unpleasant user experience.

The most likely candidate design involves a three-step process:

1. The agent’s cell calls a basic sanity check callback on the membrane proof; this can do things like check for correct structure or catch typos.
2. The agent’s cell tries to bootstrap into the network and publish their genesis entries; selected peers validate the membrane proof properly, potentially using DHT data in the process.
3. The agent detects whether they’ve successfully been granted access to the network; if they haven’t, they destroy their cell and try again with a new agent ID (and possibly a new membrane proof, depending on how the app’s membrane protection is designed).

The goal of this process is to catch as many classes of honest mistakes as possible before you try to join the network, while still allowing the immune system to identify truly bad actors and giving you a chance to try again if you’re mistakenly identified as one yourself.

So what does this mean for hosted Elemental chat?

By now you’re probably wondering whether that long-awaited milestone, hosted Elemental chat for web users, is being delayed.

Nope! This is all an expected part of the development cycle. When theory meets the real world, there’s always a process of adjusting the two to match each other. I’m glad to see it happening; it means we’re identifying all the weird edge cases that tend to hide when it’s all just theory. And as we deal with each edge case, it’s just going to make Holo-hosted apps (and Holochain in general) that much better.

Cover photo by Christian Lue on Unsplash