It’s tough to recap my experience of DWeb Camp 2023. It was an intense, invigorating, and innovative event, as was the work leading up to it. Maybe the best thing I can do is try to tell the story of the Emergence app and provide my perspective on Holochain’s involvement and learning throughout this process. So, here goes:
Intent
In March 2023 Holochain began conversations with DWeb Camp about sponsorship and offered to create a simple application for note taking and community connection (via a tag cloud and shared space for documentation). This was intended to increase the collective intelligence of the camp and to help document the conference sessions. As a local-first, peer-to-peer application it would run directly on participants' laptops and connect over the mesh network LAN that the wonderful DWeb network team sets up each year, thus not needing to connect to central servers or the rest of the internet.
At the 2022 DWeb Camp many of us experienced challenges with the scheduling app Sched. Because everyone at camp was connecting to their servers over the same low bandwidth connection, camp participants had ongoing issues loading and accessing the schedule. These conditions are exactly where distributed applications shine, and we saw this as an opportunity to give people a strong experience of a local-first application.
Tomorrow
After some conversations with DWeb’s organizing team, this initial idea grew into a plan to have our app act as a decentralized Sched and to actually facilitate a day of “open-space” style sessions, coordinating the schedule as an emergent process amongst all of the camp participants. This day became known as “Tomorrow” — a day for stepping into and embodying the technologies and social patterns that the distributed web is building towards. On that day, access to the wider Internet would be shut off so as to give all of us distributed web tool builders a real opportunity to put our tools where our mouths are! Furthermore, to double down on decentralization, during this day all the participants would be using our app for proposing sessions and coordinating in which spaces they would go.
This felt like a really exciting opportunity that we couldn’t pass up, but it came with some significant challenges. The main one being that the app would have to work on cell phones because it just wasn’t going to be feasible to have 400 people toting their laptops around the forests of Camp Navaro just to add in their sessions and browse the schedule. But therein lay the rub: Holochain is not yet cell phone ready. So, to do this, we would actually need to either get it ready for phones, or come up with a work-around that was more like Holo-hosting, where cell phone users could just use a browser to access a Holochain instance through a proxy. But of course we couldn’t use actual Holo-hosting itself because this was all supposed to work on a day where access to the wider internet was off!
The Plan
We decided to go for both options: As Plan A, we would move forward on our “zero-arc” solution for running Holochain on cell phones directly. This solution disables gossip (and thus gets around the cell phone OS penchant of killing apps that use too much CPU/Memory/Networking), but still allows each person’s source-chain to live on the phone. As a backup, in case we couldn’t get plan A to work in time (we only had a few months), we would also develop a multiplexer, a kind of “Holo-hosting-light,” that would allow browser access to a pool of Holochain conductors that we would run on a cluster of small boxes on-site. The irony was not lost on us that to get things working in this setting we would have to do a kind of micro-re-centralization of a fully distributed tool, but this is still a useful configuration of the tech to explore!
So, I did some project planning, initial app prototyping, and smoke testing for our Plan B just to make sure that Holochain could handle the load of 500 people creating sessions and doing all kinds of activities on the 3 conductors in the multiplexer which we thought would be sufficient for our little cluster. The results looked great, no load problems. We also had a nice project plan with the various aspects of all the work and our participation in the conference, spreading out the labor and ensuring buy-in across all the stakeholders, and so we gave the green light to go ahead.
The fantastic Jordan Gray took the initial design requirements and created the basic UX layout for the app. From Jordan’s designs, I built out the DNA for the app and Svelte-based single-page-app front-end. We then invested significant time in building this into a reasonably polished user experience with many, many rounds of UX/UI testing and improvements so that the app would really work well for the event and provide a powerful experience both of a distributed app, but also of the distributed coordination of a conference event. We really wanted to deliver on the ethos of the D in DWeb both at the technical and social levels.
The Details
But it turns out that there are also a couple of other challenging Ds: those of the Devil in the Details.
For a number of complex reasons, we realized about six weeks before the event that Plan A for mobile wasn’t going to land in time for DWeb. There were too many development steps with too much risk to be able to be sure it would work. Though we had the conductor compiling and working on iOS and Android, some aspects of networking were being blocked, and we realized that there just wasn’t enough time to test, debug, and get the app through the Apple/Google App/Play-Store deployment process. So we switched completely to Plan B which appeared to be working very well. I was able to build the multiplexer in a weekend, which was what we expected because it's a fairly low complexity solution. It’s just a simple Node web app sitting behind a reverse-proxy which does 3 things: 1) installs app instances using people’s keys and passwords as a seed for generating their key-pair, 2) creates a UI credential that gets stored in people’s browser as a cookie, and 3) accepts zome calls from the UI signed using that credential and forwards it to the appropriate conductor.
But then things started to get hairy: as we ran our testing sessions with more and more people, we started to get zome-call timeouts and other symptoms that looked like conductor overload. This was quite surprising as it was happening at levels of interaction well below those we subjected the conductor to in our initial smoke-testing. We soon discovered that this was because users were often just reloading their web-browsers which was dropping in-progress websocket connections to the conductor. Holochain was designed under the assumption of a single user connecting over a few websockets to a local conductor, here we were creating hundreds of unexpected websocket connections. To fix this, we added web-socket connection multiplexing into our multiplexer setup! This improved things, but still, after a while, things stopped working again. Grrrr.
So then, as we continued to debug these issues, we really started seeing the rise of additional contextual complexity. Example: our plan for deploying to our cluster using our internal nix infrastructure (based on hydra), was implemented, but incomplete because we discovered that we wouldn’t be able to serve our multiplexer service over plain http because cell phone browsers get grumpy about that and basically require https for decent user experience. So we had to include adding and provisioning ssl certificates to reverse-proxy web-servers. And well, it just snowballed from there…
What Happened Tomorrow
At this point I’m going to stop the blow-by-blow of all of the failure modes we encountered in processes, like converting and uploading data from Sched (which we wanted to do so people could take notes on the sessions all week long), or our development machines mysteriously taking forever to deploy data to the cluster because they were actually connected to a very slow Wifi when the Ubuntu UI showed that they were on the Ethernet, etc. Each one of these issues was fixable. Each one took minutes or hours to address. Time and again, we had successful test-runs followed by complete failures.
When we got to DWeb Camp it looked like the app was finally ready, but then this saga continued all throughout the build days, and even into the two days leading up to Saturday’s Tomorrow day of emergent sessions. We did, finally, get it working. Bugs at all levels, from the UI to the network configurations, to Holochain itself, were squashed. However, by this point, the DWeb conference team had already made the call to use the backup analog process for conference scheduling (which was a more than reasonable decision given the challenges up to that point), but we were still going to use the app for its initial purpose of note-taking and sensemaking. So, there we are with the whole cluster provisioned and hundreds of people starting to use the app when suddenly the power went out to the whole valley. This meant that though peoples’ phones were operating, there was no local LAN to connect to because the Wifi access points were down. It was a perfect example of an unending series of cascading failure modes, and an ideal embodiment of exactly what the Tomorrow day was supposed to be a test of: how fully decentralized can our tech stack be? Not just as we talk about it, but in actual practice, in the middle of a camp in the redwood forest.
On the flip-side, this led to a beautiful experience in further emergence. We had planned to do a get-your-hands-on-it session for people to experience running various Holochan apps. Well, that actually worked! Because even though the Wifi access points were down, we had the idea of turning on a cell-phone’s hot-spot functionality which, even without cell service, made it an ad-hoc Wifi router for Holochain nodes on laptops to connect in the middle of the woods with no power! This felt like a significant real-life demo of the power of local-first distributed computing architecture.
Lessons in Complexity and Teamwork
So in the end, what did we learn? Well, there were many lessons, small and large, which we are integrating back into our work, process, and code. Part of the difficulty of expanding contextual complexity comes precisely from the fact that it is invisible at the outset, even when you account for invisible additional complexity, which we did. We certainly detected and fixed a bunch of subtle performance bugs in Holochain. And we greatly increased our in-house capacity to build out apps with decent user-experience.
One of the most powerful lessons for me, however, was the lived experience of what an amazing team we have assembled. As each one of the problems we encountered arose, someone jumped forward to help figure it out, or work on it until we moved through to the next challenge. I find that our team’s collaboration isn’t just about solving the problem, but the grace and joy of working together in doing so.
I also want to personally (and on behalf of Holochain), thank the DWeb community and especially the on-site network team. Your tireless work was inspirational as well as being essential to everything we did. It was a pleasure to work with you.
The invitation of DWeb’s “Tomorrow” to take risks and bring technologies to use rather then just talking about them was answered by Holochain and others. It served as a forcing function to put our tooling and teamwork to the test in a setting where the contextual complexity was quite high. With Holochain in beta, this experience allowed us to squash some very difficult to find bugs. We will continue to prioritize partnerships and projects that put Holochain to the test. And if anybody is interested in using a very cool app for helping run open-space/unconference style gatherings, please get in touch with us!
Special thanks to Eric Bear (process design, testing, holding the relationship with DWeb), Jordan Gray (initial spec), Aaron Brodeur (UX/UI), David Braden (multiplexer & app stress testing, tooling, and Holochain debugging), Jost Schulte (testing across all levels), Stefan Junker (network infrastructure and nix deployment), Gregory Jensen (Holochain testing and debugging), and the many others who put time and labor into this project. Similarly, thank you to everyone in the Holochain team and community who showed up to the repeated test sessions both before and at DWeb Camp.
