How a green energy startup adopted a technology, found a tribe, and recaptured a treasure.

A call to adventure

In June 2018, Mike Gamble found himself at a meetup in Melbourne, Australia with a handful of people—fewer than he could count on both hands. Mike, an affable gentleman with a Canadian accent and a veteran of numerous complicated and important sounding engineering projects, was here because of his friend Adam Thompson. You see, Mike was planning an Ethereum-based crypto venture with his friend Sim Wilson. But Adam had been bugging him for months about a new distributed ledger project called Holochain. Mike thought he’d at least learn what this thing was about.

Sitting beside him was Philip Beadle, a plain-spoken, rough-shaven code wizard from the Holochain dev team, along with colleagues David Meister and Willem Olding. During the course of the meetup, Philip proceeded to fire a small volley of terse cannonballs at blockchain’s battlements.

“Scalability.”

“You don’t need consensus.”

He let the pungent smoke of his words’ impact hang in the air. It was enough to arouse Mike’s curiosity. He joined the inaugural Holochain dev camp a few months later, during which he pressed Philip for technical details like “can this thing run headless?”

“Absolutely.”

Mike was starting to get pretty interested by this point. He gradually stepped away from the Melbourne blockchain scene and started nurturing a community of Holochain enthusiasts. He persuaded Sim, now the CTO of a startup called RedGrid, to check Holochain out. RedGrid was building an ‘internet of energy’ ecosystem with the intent of accelerating the world’s transition to clean electricity, and Mike thought Holochain might be a decent match for their unique needs. Especially attractive was the fact that three of Holochain’s devs lived right there in Melbourne; the prospect of local support was a big plus for him.

Somewhere along the line, Mike found himself at the helm of RedGrid’s engineering team. As they began to make connections with the Holochain team and supporters, Mike felt like they were joining a movement rather than adopting a technology. He described a weekend intensive of Holochain application developers on the Sunshine Coast as “historic.” He would later say that

“there is a blockchain industry, but a Holochain community.”

And as a fan of biomimicry, Mike recognised Holochain’s efforts to emulate the natural world—specifically its ability to coordinate activity among massive numbers of autonomous agents without a central controller. In principle, he believed, this thing should beat client/server for resilience and blockchain for performance.

In principle.

Reality bites

Fast forward a year and a half. Monash University had been working with RedGrid for six months in service of Monash’s Net Zero Initiative—an ambitious project to transition all of their campuses to net-zero carbon emissions by 2030 and share their lessons with the world.

Together RedGrid and Monash were developing a Holochain-based transactive energy market (TEM) app to power the campus microgrid. A TEM connects an ecosystem of energy consumers and producers and allows them to tender and fulfill contracts to supply electricity. These markets, such as the Australian Energy Market Operator (AEMO), are poised to become a common feature of national energy grids.

Along with the consensus of the power industry, Monash and RedGrid recognised that the grid of the future will be decentralised, decarbonised, and digitalised. Decentralisation creates a more robust, adaptive grid, reducing the impact of disruptions and giving local communities more autonomy over their energy provision and distribution decisions. It demands communication and orchestration between individual components in the system, which is why digitalisation is important. But that creates a challenge: with a zillion devices all talking to each other, there is a lot of data being produced, and somebody’s going to have to process it.

That’s why both Monash and RedGrid believed that digitalisation would have to be decentralised too, not just energy production and distribution. Their convictions echo the computer industry’s growing sense that, with more devices coming online, intelligence will need to be pushed from the centre to the edges in order to keep up with all that data. It was RedGrid’s job to prove that Holochain was the right choice for this job, starting with the TEM component.

The rubber hit the road in July 2020 with an invitation from the Monash Net Zero people: “Let’s see a demo.” Initially the demo called for a benchmark involving five nodes participating in a market. Here’s how it was to work: When an energy-consuming node (say, a campus building) wanted to buy energy, it would open itself to receive bids from a marketplace of energy producers (say, on-campus power generation equipment or national grid companies) for a future time period. These producers would respond by generating bids for slots they wanted to provide power for, writing them to their data stores, and privately communicating them to the consumer. On receiving each bid, the consumer node would record it in their own data store. And periodically (once or twice a second) they would write a ‘market plan’ entry consisting of all the bids they’d received so far. Lastly, the final market plan would get shared with all the bidders.

Because the design was to be used in a distributed network of entities who could be consumers, producers, or even both at any given time, it was expected to be a good test of Holochain’s suitability for such a demanding edge computing use case. Each market cycle involved a lot of messages and writes, so it was also a good test of throughput and latency. Both were important metrics for performance; latency doubly so for ensuring that the consumer node awarded contracts fairly.

‘In principle’—there’s that hope again—Holochain’s completely parallel design would allow all this to happen at lightning speed. Then Monash said, “hey, let’s make it 30 nodes instead of five, just to see what it can do.” And then they pressed play.

It must’ve been a discouraging moment for the RedGrid team—their faith in Holochain, their many person-hours of work, their close collaboration with the Holochain ecosystem, culminating in a mere single bid every few minutes. (That’s roughly 0.005 transactions per second for you blockchain folks—although that’s a bit of an inaccurate comparison; read the epilogue to find out why.) One bid even took 17 minutes to go through.

On top of that, Monash noticed that these nodes had to talk through a ‘switchboard’ server called sim2h. That centralisation point was a show-stopper (and probably partly responsible for the low throughput and high latency).

A glimmer of hope

Relying on the relationships that had already been built, Mike reached out to Arthur Brock, co-founder of the Holochain project. Arthur introduced Mike to Holochain RSM (Refactored State Model), a ‘skunkworks’ effort to rewrite Holochain from scratch. (You see, our devs at Holo, primary sponsor for Holochain development, had been having our own troubles testing the prototype of the HoloFuel cryptocurrency.) Arthur promised that RSM, although not heavily benchmarked, should be a few orders of magnitude faster than the previous version.

Once again, the RedGrid team took a leap of faith and decided to push ahead with Holochain RSM. Mike and his colleagues were invited into a close-knit group of early-access app developers and Holochain core team members. Mike was pleased to find people from the community rallying around them, working around the clock (literally—they spanned four time zones) to help them migrate the TEM code to RSM.

Guillem Córdoba and Connor Turland, two of the community devs that RedGrid contracted, appreciated how easy the migration process was. They said the new SDK was more straightforward, and it helped them cut the number of lines of code by half.

But they also had to circumvent issues in various layers of RSM’s immature and rapidly changing feature set. A big challenge was the lack of working network transport code, which meant that communication had to happen through node.js-based middleware. But they made it work. By late August RedGrid was once again ready to demo TEM for Monash.

The treasure is recovered

You might say the results were slightly different. The market node was able to receive and store messages simultaneously from 29 bidder nodes to the tune of 87 bids per second without breaking a sweat. This was a ~15,000× improvement—five orders of magnitude.

Per-transaction latency went down too. In the previous demo, most of the slow transaction speeds were caused by latency in various parts of the system. Now data moved around so quickly that it was difficult to even capture latency numbers.

Needless to say, the Monash team were rather impressed. They even hinted that they might drop the streaming message bus they were using elsewhere (think Apache Kafka) in favour of Holochain.

Strength in a band of allies

Aside from the performance improvements, one thing that impressed Monash was the speed of RedGrid’s turnaround. Mike once again credits the developer ecosystem. In an interview with Holo director David Atkinson, he reiterated,

“I should point out that what attracted me to Holochain was the support, not so much the tech. And it’s just in the last month that this has proven out, with Guillem and Connor [and Alfredo, Jesús, and Eduardo] helping us… You can see in GitHub and Discord their discussions with David Meister and Tom Gowan [Holochain core devs], to the point where a community dev actually found things and updated the core. It’s really pretty impressive. And Monash recognised that.”

Dr Adam Bumpus, RedGrid’s CEO, agrees:

“And that was really cool, because you’re building together. For us as a company, we’ve had lots of good support from Art [Brock] and Eric Harris-Braun [Holochain co-founders] from the beginning, and obviously other people Mike mentioned... It makes us feel good—about the community, about the whole mission, and also there are other people out there who are not building on Holochain but believe in what we’re doing, the Holochain/RedGrid combination.”

For us in the Holochain team, this is encouraging news. It means that a thriving ecosystem of developers and supporters is coming to life around us. This feels fitting, because Holochain is modeled after the coordination networks that emerge in living ecosystems. And in emulation of our technology, we’ve tried our best since the beginning to nurture rich human relationships. These relationships, we believe, are more valuable in the long run than flashy marketing campaigns and shallow partnerships.

If your startup is courting a potential partner, one of the first things they’ll look at is how well supported you are. If others are walking alongside you, championing your vision and helping you out, you have a better chance of weathering the inevitable storms. It’s all about resilience, and resilience is attractive.

We’re glad that such a meaningful project as RedGrid has joined the Holochain ecosystem, and we couldn’t be happier to witness the ways the ecosystem is supporting them. And we’re seeing this again and again; the people who’ve chosen Holochain support each other. Community-led dev camps, meetups, documentation efforts, contributions to Holochain Core, open-source library collaborations, chat conversations, and plenty of healthy feedback and critique on Holochain itself are making the ecosystem much stronger than our small team ever could by itself. Just like RedGrid, we feel like it’s making Holochain resilient and well-equipped to take on the challenges of bringing a new way of building software into the world.

Very long epilogue: A deep dive into the numbers

I think the big story here is how encouraging it is to see an ecosystem of creators and builders supporting each other to help bring one player’s vision to life. But the numbers are interesting in their own right, to show just how much Holochain RSM has improved over the previous version.

So first I’d like to give some more details about the test as I understand it from talking to Mike, Connor, and Guillem. (Sorry to the three of you if I get some details wrong!) As I go along I’ll try to interpret the results for you.

Each market period spans nominally 24 hours, and bidders can bid on as many five-minute intervals in that period as they like. Each bid consists of about 1 kb of data. Because the consumer node has to record each bid it receives, the activity in the bidding window’s time period is a good test of throughput, or write ops per second. (In the blockchain world this would get called transactions per second, or TPS, although this isn’t a terribly meaningful metric to apply to Holochain—more on that later.) None of the entries are public, so they don’t have to be published to the DHT, but they are written to their authors’ personal journals and broadcast to the appropriate parties. This is where latency matters. And finally, the market plans get quite large—by the end of the test, the final market plan is about 100 kb. This tests Holochain’s ability to write large blobs without choking.

Monash didn’t want to artificially cripple this test in any way; they wanted RedGrid to hold Holochain’s feet to the fire in order to get a sense of how it might perform in the real world. (Mike tells me that a marketplace network might consist of tens of thousands of nodes, each of whom can be both producer and consumer simultaneously.)

In the real world, these nodes would be autonomous computers running in appliances, energy generation/storage hardware, buildings, or energy companies’ offices; for the purpose of the demo it was enough to use multiple Docker containers on one physical machine (and besides, RedGrid weren’t able to directly access the devices in Monash’s lab during one of the world’s longest-running COVID-19 lockdowns). That means that wire latency wasn’t being tested, just latency of the system itself. Additionally, because the team had to use node.js middleware in place of the not-yet-implemented Holochain P2P network transport, the speed of the middleware contributed to the latency figures. (Mike suspects that it added latency.)

Here’s a graph of how the consumer node fared in the RSM demo. It doesn’t capture initial spin-up of the application (which causes a momentary CPU spike to 100%); it just captures the consumer node’s ability to keep up with all the bids it’s receiving.

Each bidder is throttled to send only three bids per second. 29 bidders × 3 bids/sec = 87 write ops per second; the consumer node is able to handle everything the bidders can throw at it.

The initial spike in activity is accompanied by a modest CPU spike of only 40%. A big reason for this improvement over the previous version of Holochain is that the core team has put a lot of work into improving concurrency, removing bottlenecks in places where threads were waiting for other threads to finish. (When testing HoloFuel on the previous version of Holochain, we found that this problem compounded when you introduced multiple nodes.)

RAM also stays nice and low. Optimal use of the database engine (LMDB) is responsible for most of this, and the core team is constantly watching how they use and tune LMDB to keep a good balance between memory consumption and disk swapping latency.

We see ops/sec tapering off as the test continues, yet CPU and RAM don’t appear to be bottlenecks. The RedGrid dev team suspects that this may be due to unsynchronised bidding times—that is, bidders start at roughly the same time but some finish early.

I’m eager to follow RedGrid’s future testing efforts. I asked them why they throttled bidder nodes. They explained that, when bid frequency increased, the consumer node hit a bug that caused it to go to 100% CPU and hang. A fix is already in the works and will be committed soon; It’ll be interesting to see what test results will look like at that point (my prediction is that we’ll see another >5× improvement). Guillem also shared with me that architectural changes to their TEM app, such as relying more on the DHT, might improve throughput by an order of magnitude as well.

Apples and lobsters

I’d like to talk about what things would look like if RedGrid had built the TEM app on some other system—traditional cloud, Ethereum, IOTA, or DFINITY, for example. But it’s not an apples-to-apples comparison; it’s more like apples-to-lobsters. Every time I think I’ve captured the essence of each platform such that I can compare it to Holochain, another dimension pops up and I have to rethink everything.

For instance, that old favourite metric of blockchain enthusiasts, transactions per second, doesn’t really apply to Holochain. Because each participant carries their own weight in parallel with other nodes, there is no max TPS for the network; the equivalent metric would be max write ops per participant per second.

I could say a lot more about TPS and other metrics. But for a developer it always, always, always boils down to the question “is this technology appropriate for my use case?” I hope to use this question as a guide in future writing, using RedGrid as a case study to compare Holochain with other technologies.

Are you interested in this subject? Are you building a project and trying to decide what technology to build on? I invite you to reach out to me on Twitter and suggest other technologies or use cases that you’d like to learn more about.