Summary

As more and more developers get excited about Holochain and start building real things with it, we find ourselves spending more time on refining the developer experience. We’re eradicating performance bottlenecks, improving tooling, and adjusting APIs to become more pleasant to use. We thank each and everyone one of them for lighting refinement fire with their creativity!

Highlights

  1. Developer Experience: The Details Matter
  2. Performance: PickleDB Storage Implementation Speeds Commits By 92%
  3. Tooling: New Options Added to `hc keygen`
  4. Breaking Change: API Deprecated RPC Format Removed
  5. Breaking Change: `JsonString::from()` Doesn’t Work With Strings
  6. Logging: Details In Failed WASM Executions
  7. Cryptography: New Features Enable DPKI Apps
  8. Lessons Learned: ‘Swarming’ Gets the Job Done
Holochain Dev Pulse April 1–April 8th, 2019

Details

1. Developer Experience: The Details Matter

If you’ve been following our project for any amount of time, you’ll know we’re not interested in crypto hype.

We’re dedicated to building a framework and protocol that will help real developers create real things that will enable forms of social cooperation that the world has never seen before. That’s why we’ve always been focused on developer experience.

Little details matter, and our goal is to make Holochain a framework that is a pleasure to use.

In the early days of our refactor to Rust, Holochain wasn’t yet useable and things were pretty rough around the edges — today, however, we can say with confidence that things are improving week by week. Our documentation project is nearly complete, we’re spending lots of time educating developers, and we’re receiving feedback on what doesn’t work. The most gratifying thing about it is that it means that people are beginning to build in earnest, even before all the features are complete. We urge you to keep pushing us to improve — join our Mattermost App Dev Chat (email help@holo.host for a registration link), post an issue on GitHub, or even submit a pull request if you’re feeling adventurous!

Performance is one of those areas that isn’t important at first, but becomes critical once people start to use your software. We reported a big speedup last week, and have another this week. We’re continuing to dig into bottlenecks, so expect even more improvement soon.

We’re also working on core dev experience, turning our regular release process into a fully automated one. You probably won’t see much of a change as a result, but it frees up developer time and increases the consistency of binary releases.

Developers Working

2. Performance: PickleDB Storage Implementation Speeds Commits By 92%

Sometimes abstractions complicate a design unnecessarily. Developers are familiar with the YAGNI principle. But sometimes they pay immense dividends.

Early in the refactor to Rust, we recognized that Holochain would be used in a lot of different environments. We therefore took the time to abstract out the parts we thought ought to be replaceable, like the storage layer. There have been two storage implementations thus far — an in-memory implementation used by the test and dev conductors and a flat-file implementation used by the production conductor (and optionally, the dev conductor).

We’ve just finished implementing a new, super-fast storage implementation based on PickleDB, which is an in-memory, key/value store with automatic persistence to disk. PickleDB persists to disk on shutdown and will need to be configured to persist immediately on writes.

We chose this library because it introduces no external dependencies and was easy and fast to integrate.

Benchmarks!

test bench_file_eav_many_to_one … bench: 168,120,460 ns/iter (+/- 121,749,965)

test bench_file_eav_one_to_many … bench: 192,197,600 ns/iter (+/- 104,684,083)

test bench_pickle_eav_many_to_one … bench: 12,436,580 ns/iter (+/- 6,530,620)

test bench_pickle_eav_one_to_many … bench: 12,237,040 ns/iter (+/- 6,906,985)

There’s been a 92–95% decrease (roughly) on these benchmarks.

3. Tooling: New Options Added to `hc keygen`

`hc keygen`, the keystore generation tool, has two new options:

  • ` — path`, which lets you specify the location of the generated keystore.
  • ` — quiet`, which suppresses all output except two lines — the public key and the path to the new keystore file.

These, along with piping the passphrases via `stdin`, are predominantly to support scripting for Holo Host and our automated build system. That said, you might be able to find some clever uses for them!

Breaking Change Reminder: The keystore file format changed in 0.0.8, so you’ll need to regenerate all your key bundles with `hc keygen`.

4. Breaking Change: API Deprecated RPC Format Removed

In Dev Pulse 19, we briefly mentioned that the format of JSON-RPC calls was changing; the method name would now simply be `call`, and the instance, zome, and function names would move into the params of the request. We kept around the old method for a while, but it has now been removed. If you’re using hc-web-client in your GUI, make sure you upgrade to version 0.1.3 or newer, and switch all usage of `call` to `callZome`. Read the full documentation for hc-web-client. If you are using hc-redux-middleware, make sure you update to the latest version 1.1.1.

5. Breaking Change: `JsonString::from()` Doesn’t Work With Strings

Previously, you could convert a string into a `JsonString` using Rust’s standard `From` and `Into` traits. It would assume that a string was already JSON, which would cause problems if the string was not bounded by double quotes. We’ve removed that trait and moved its behavior to a method called `JsonString::from_json()` to make the intent more explicit. So please convert all usage to `JsonString::from_json(“{\”hello\”: \”world\”}”)` if your app DNA has any code like:

let a = JsonString::from(“{\”hello\”: \”world\”}”);

or

let a: JsonString = “{\”hello\”: \”world\”}”.into();

Pro Tip: Did you know you can automatically convert JSON into one of your DNA’s native entry types using:

let foo= Foo::try_from(hopefully_json_string)?;

This saves keystrokes and is a bit more explicit when compared to using Serde’s `from_str()` method. It’s useful when you’re receiving JSON from a `call()` to another zome or a bridged DNA and you don’t have the convenience of `hdk::utils::get_as_type()`.

6. Logging: Details In Failed WASM Executions

We’ve been getting reports of “Argument deserialization failed” errors when zome functions are executed. This can happen when the JSON passed to a function doesn’t match the expected parameters and types. This is annoying to debug by hand, so instead, you’ll now get an error message telling you exactly which zome function failed and what arguments were passed to it. Again, we hope this will be yet another incremental improvement that makes your development experience better.

7. Cryptography: New Features Enable DPKI Apps

Last week, we shared a bit of our vision for a decentralized identity framework — Distributed Public Key Infrastructure (DPKI). As we’ve written in the past, to design this well, we need to consider both technological and social factors. Self-sovereign identity is both a new idea and an ancient pattern — formerly, our identities were fluid, linked to our relationships with our peers. As societies grew, we depended more and more on centralized guarantors of identity like birth certificates and passports.

With public key cryptography, we now have the technological tools to build trustworthy digital identities based on others’ acknowledgement.

The missing piece is creating a way to make it accessible to everyone.

We are building a reference implementation of the DPKI concept called DeepKey. It’s an app on Holochain that will enable us to manage the public expression of our identities. It will let us link public keys across all devices, revoke the privileges of lost or stolen keys, and have our ‘root identities’ signed by third-party guarantors.

DeepKey Key Management Interaction Flow

We don’t think that DeepKey will necessarily work for everyone, however. We’ve therefore decoupled it from Holochain so that you can write your own compatible DPKI implementation — or even run in ‘standalone mode’ without any DPKI implementation. In order for a user to take advantage of DPKI, their Conductor needs to be told which DNA bundle it should use as a DPKI app, bootstrap the DPKI app, and ask the DPKI implementation to generate a new agent key for each new app instance it creates.

All three of these abilities are now implemented in the Production Conductor, along with all the supporting cryptographic tools.

8. Lessons Learned: ‘Swarming’ Gets the Job Done

As we approach broader rollouts of the Holo Closed Alpha testnet, we’re being called upon to fix bugs at lightning speed to unblock the Holo dev team. We’ve started ‘swarming’ around issues — a few developers jump into action, along with the original bug reporter, and tackle the issue in a focused session. Multiple sets of eyes all witness the problem and scan the code together, looking for the cause and debating solutions.

For example, last week, a Holo dev was having an issue with DNA caching. When they recompiled DNA packages, the Conductor wouldn’t pick up on the updates if it had already executed any of the zome code in the DNA. As with most caching issues, this was causing debug frustrations due to the limitation of the invalidation logic. The core dev team swarmed with the bug reporter and realized that the issue was quite different from what they expected.

We’re finding that sanity checks like this are becoming incredibly helpful for zeroing in on the real issues. Even more valuable is sanity checks that one that asks, “is this feature/solution solving the actual problem, or do we need to spend more time understanding the dev’s core need?” It’s easy to assume we understand what’s being asked and go race off to build something, but deep listening is giving us a much better understanding of the issues — which results in a true solution.

Development Status:

Install Holochain Learn about Holochain Join the Discussion