TL;DR

Sim1h is a new way for devs to build, run, test and get user feedback on their hApps before lib3h is done. It also enables us to work in parallel with a full Holochain stack on small chunks of lib3h and then plug them in as soon as they’re tested, debugged, and ready. Scroll down to see how easy it is to get started with sim1h.

The Details

The cat is out of the bag. Over the last 3 weeks (David Meister created the first commit during our dev retreat in Montreal) we have built a near-term, centralised networking component for testing called sim1h. It is a simulation of lib3h, our p2p network and DHT implementation that has been—and continues to be—the main focus of the Holochain networking team.

When I first heard Jamison (our Director of Product) suggest implementing sim1h alongside lib3h, of course I thought: “WTF?! A centralised back-end for Holochain? ”. I also felt myself rejecting this idea because I thought we needed all hands on deck to iron out lib3h. But then two insights in particular made me shift to the other end of the spectrum and feel super excited about this idea:

  • Debugging and analyzing test networks: Imagine lib3h were all ready to go and we enter the Open Alpha Testnet with a fully decentralised network stack. Even if the whole Holochain stack worked without any bugs and quirks (though we do expect some bugs—that’s why it’s called alpha), there would still be lots of hApps (both our internal as well as external hApps) that will need debugging.

    Exactly for this reason we were planning to have Holochain nodes and HoloPorts send logs and other state information to a single node for diagnostics anyway. Sim1h just takes this idea to another level, and reduces work, since it is running as a simulated network that lets us see all the information that nodes are exchanging without having to implement special debugging and analytics code. Usually, distributed applications don’t let anyone see all information being exchanged on the network, but this is crucial for accurate and quick debugging.
  • Iterative, parallel staging: During these last few months we’ve become more aware of the hidden complexities involved with trying to integrate two horizontal pieces of a vertical technology stack. Holochain core, with it’s overlay network of AgentIDs, is working fine within the boundaries of its tests and in-memory network. Lib3h is working fine within the test suites that try to mock a real-world usage. However, putting two built-out and complicated pieces together after developing each in isolation led to all sorts of complex behaviors that take quite some effort to analyse, test out, and debug. Eric described some of this in a recent AMA.

    Having a much less complex replacement network implementation that simply works as expected gives us a fully working Holochain stack such that hApp developers can start testing and using their hApps. I am personally even more excited about the evolutionary path this enables because we can now take and replace pieces of the whole system in small iterations. For example with sim1h we can  tackle Holochain’s direct messages in a peer-to-peer manner, while the DHT gossip and queries still run through the simulated network. And then as a next step we’ll be able to easily pull in lib3h’s DHT code into a single simulation and prove that the high-level logic and our new DHT algorithms work fine (without implementing a full peer-to-peer DHT - and testing that simultaneously). And we can do all of that, before adding the complexity involved with low-level connection handling, NAT traversal, peer discovery, etc. This provides huge improvements for us in terms of time-to-delivery and testing capabilities.

So what can I do with sim1h right now?

As mentioned, we implemented sim1h over the last 3 weeks and it is already working. As of last week, all our Holochain app-spec tests (190 assertions in 30 scenarios) are passing when using sim1h as the network back-end. As of this week, we have started proving out multi-peer activity in our internal hApps using sim1h, as well. And you can do the same with your hApp right now!

1) Select sim1h as Network Type

Since Tuesday's Holochain release v0.0.32-alpha2, sim1h is built-in and all you need to do is select it as the network type in the conductor config:

[network]
type = "sim1h"
dynamo_url = "http://localhost:8000"

2) Add URL to Parameter

The only parameter it needs is the URL to the Dynamo database. You don’t need to have that hosted inside AWS. The nix-shell derivations inside holochain-rust as well as sim1h include two new commands: dynamodb and dynamodb-memory which both start a local instance and can take a -port parameter. You can run those commands either on your local development machine to exercise multi-agent tests (what we do for Holochain’s test suite) or run them on a test server in a LAN or VPN.

3) Set Two Environmental Variables

The only other thing that is needed is to set two environment variables when starting the Holochain conductor:

AWS_ACCESS_KEY_ID
and
AWS_SECRET_ACCESS_KEY

These get picked up by Rusoto, the AWS client library we’re using in sim1h to connect to Dynamo. If these are not set, Rusoto will bail and not even try to establish a connection (which will be visible in the conductor logs).

If you run a local Dynamo instance outside of AWS, it doesn’t have a way of checking AWS credentials and will allow any access key to connect to the database. This is why you want to do this only over LAN or VPN. Exposing a self-hosted, unsecure DynamoDB instance is a bad idea.

But you still have to set those two variables for Rusoto to be satisfied.

While the secret really doesn’t matter, it just has to be set. That’s because the access key ID actually makes a difference because it acts like a namespace. A different access key ID will see different tables.

So, this is how you would invoke the conductor, for example:

AWS_ACCESS_KEY_ID=HoloCentral AWS_SECRET_ACCESS_KEY=.. holochain -c /etc/conductor-config.toml

If you have multiple teams using the same intranet server, you can have them use different access key IDs to not run into collisions, like this:

AWS_ACCESS_KEY_ID=hApp-Team AWS_SECRET_ACCESS_KEY=.. holochain -c /etc/conductor-config.toml

Lastly: Inside the database

Inside the database you will currently find one table per DNA. Using the database with a new DNA will create a table with that DNA’s hash as table name. Entries and their aspects are stored under their hash as the document name. Sim1h simulates direct messages by maintaining inboxes for each agent. In order to view this data, any DynamoDB client can be used, like for instance dynamodb-admin. Please note that, again, the AWS_ACCESS_KEY_ID needs to be set accordingly. Also, the region has to be set to (the hard-coded string) “holochain-testing”:

AWS_REGION=holochain-testing AWS_ACCESS_KEY_ID=hApp-Team
AWS_SECRET_ACCESS_KEY=.. dynamodb-admin

That’s it! We would love to hear any feedback you have on how this is working for you. We’ll be sharing our next steps as soon as next week.

Nicolas Luck
Lead Holochain Developer