Summary
This week, the focus was largely on cryptography. More and more of our long-term work providing cryptographic support to applications is stable and being incorporated into our official releases. We’ve also started the important work of perfecting performance profiling and tuning.
Highlights
- Data Integrity/Identity: New KeyStore Ready — Available to Apps and Core
- Performance: Optimizations!
- Network Bootstrapping: ‘P2P Ready’ Message
- Debugging: Logging Flag for `hc run`
- Lessons Learned: Closing the Gap Between Core Devs and App Devs
Details
1. Data Integrity/Identity: New KeyStore Ready — Available to Apps and Core
Identifying yourself on the Internet is a hard problem, currently solved by centralized PKIs like certificate authorities — doing it without centralizing power is an even harder problem.
However, we like hard problems, so we set out to create a distributed public key infrastructure (DPKI) that puts you at the center of your digital identity, while simultaneously solving many of the user-friendliness problems of decentralized trust.
Over the past couple of months, a good deal of our work has been related to this in some way or another. From low-level cryptography support, to the DeepKey App, to the Holo key generation sequence — this is a large project that spanned all of our dev teams. A lot of the work has already landed, but we still have more to do. This week, we rolled out changes to our KeyStore Manager that stores all your secrets, your private keys, and key seeds. They live in secure memory buffers in the Conductor, which performs cryptographic operations on your behalf.
These changes allow us to realize all of the functionality of DeepKey, which is our default implementation of the DPKI pattern. We intend DeepKey to be replaceable, so you can write your own implementations to better suit your community’s needs.
Your device’s secrets are used by the Conductor to sign data and control access to the RPC interface, the N3H networking layer to encrypt communications, and the apps themselves. You’re already able to sign and verify signatures within your zome functions, but now you can generate new seeds and keys and specify which key to use for signing. Because you don’t have direct access to any of the secrets, you’ll refer to them by their string ID.
We’ve currently implemented HDK functions for signing, signature verification, and secret generation. We’ll be introducing encryption and decryption functions in the future.
New functions in `hdk::api`:
- `keystore_list()`: Return the IDs of all secrets in the key store.
- `keystore_new_random()`: Create a new root seed and store it at the specified ID.
- `keystore_derive_seed()`: Create a new seed from the specified seed and store it at the specified ID.
- `keystore_derive_key()`: Create a new key pair from the specified seed, store it at the specified ID, and return the public component.
- `keystore_sign()`: Sign a payload using a specific key and return the signature.
- `sign_one_time()`: Create a new random key pair, sign a payload, shred the private component, and return the signature and public key.
Key Management Part 1 and Part 2.
Breaking changes: Any keys created with `hc keygen` will need to be regenerated.
2. Performance: Optimizations!
Our Holo dev team recently demoed full, end-to-end networking. It was very exciting to log in and use a Holochain chat app running on a public website; but as more testers joined, it got pretty slow. Each message would take a minute or more to send. This complaint was echoed by devs in our recent Sunshine Coast Holochain Intensive.
After some performance tracing, we identified the cause as an inefficiency in the networking handler, which communicates messages between Holochain and the networking layer. When it receives a message, it needs to figure out for which DNA instance it intended. Unfortunately, it was re-hashing the DNA for every message — and there were a lot of them. The good news is that we managed to fix this issue pretty quickly. We also removed some redundant filesystem reads from our EAV implementation. The end-result looks way better:
We’re now eager to track down more performance issues, such as slow persistence implementations and high CPU usage when the Conductor is idle. This is a good place to be in, because not only do we now have the time and tools to do it, but we have a reason as well. Real apps are uncovering real bottlenecks and forcing us to remove them. Optimizations like this are no longer premature.
3. Network Bootstrapping: ‘P2P Ready’ Message
Bootstrapping a Holochain app network is currently a do-it-yourself process. We don’t yet support mDNS discovery or supply a bootstrap server like we did in Holochain-Proto. This will change, but for now, you have to start one node up as a bootstrap node and point all other nodes to it. Previously, the Conductor would output a friendly copy-and-paste URL, but it was hidden behind a log-level flag and tended to get lost in the debug stream. Now, you’ll see a message like this whenever you start the first node’s Conductor:
Network spawned with bindings:
- ipc: wss://127.0.0.1:56175/
- p2p: [“wss://192.168.0.28:49210/?a=HcScJkw6MgzKpewaa8aot4DT45qWSzo8zC3ei6yAA3GOdzfbVhj9O544zB6ODua”]4. Debugging: Logging Flag for `hc run`
In recent versions of Holochain, we were debugging concurrency issues in the networking components. We ended up turning on a firehose of debug messages for the development Conductor, which is inconvenient for app developers. Now those messages are suppressed by default, but you can turn them back on for `hc run` with the ` — logging` flag.
5. Lessons Learned: Closing the Gap Between Core Devs and App Devs
As a core developer, your role is to build the foundation of what makes Holochain work. This requires you to keep a fairly complex world model in your head as you reason about what’s next. It all also needs to connect in the context of the wider world — your target users and their needs — so it’s a bit of a juggling act.
A couple of core developers shared their experiences with recognizing areas where they weren’t receiving the right context. They said that as app development ramped up, they received more feedback on bugs and desired features, but the relative urgency of some of it wasn’t understood clearly.
They also shared their renewed appreciation for how important it is to close that communication loop — to get clarification about what’s being asked and why, get more details than needed, and investigate timelines and blockers.
The core devs also had an opportunity recently to interact more closely with app developers in the wild and within our own team. This has given us valuable insight into devs’ current pain points and how they’re using and experiencing Holochain. We also have a few team members who have formed an informal task force to champion a satisfying app developer experience. We’re expanding in all directions right now, so some changes will come slowly. Developer experience has always been one of our top priorities, so we encourage you to keep us accountable regarding this goal!
What are the specific sticking points that bug you most when developing on Holochain? Share your comments on the Holochain App Dev Channel on Mattermost.
