Roomy Deep Dive: ATProto + Automerge
Hey folks! We just released our first group chat Demo on ATProto with Roomy!
You can now create and join public group spaces with channels and categories. Everything is still very work-in-progress, but you can login and test it out now!
Today I want to do a deep-dive on how Roomy works. Warning, this might get out-of-date pretty soon, we're moving fast!
CRDTs
Roomy is built on something called Conflict-Free Replicated Data Types, or CRDTs. That means that when somebody chats on their computer, possibly while offline, and another person chats on their phone, while also offline, they can still get back online and sync up with each-other later.
The CRDT makes sure that after they sync with each-other, they will both see the same data in the chatroom. It automatically combines all of the changes in the chatroom without conflicts.
We use the CRDT provided by the excellent Automerge library, made by the Ink & Switch lab as part of the local-first paradigm they’ve championed for the past decade.
ATProto & The PDSes
Roomy is also built on ATProto, the open protocol powering Bluesky. In ATProto, each user has their own Personal Data Store, or PDS, even though they usually don't have to think about it.
When you make a post or a comment, or even like another person's post, it creates a new record on your PDS, and that gives you some level of "ownership" over your data. While Bluesky offers PDSes for you, you can also host your own PDS, to really take control of your data, and this is important for preventing Bluesky from "capturing" that data in the future.
Roomy stores all of your data on your PDS, too. It also stores all of your chat data in your browser window, and allows you to continue to "chat" while offline. It will synchronize your chats later, when you get back online.
Because the chat data includes your chats, as well as everybody else's, it acts as a backup of your whole chat history. If you were to delete all the data from your PDS and you still had your browser login, it would actually replicate the data from your browser back to your PDS.
How We Store Data On the PDS
In Automerge everything exists inside of different "documents". The merging of changes all happens inside of a document. Right now each direct message is its own document, and each public space is its own document.
When you save an Automerge document, you export what is called a "snapshot". That snapshot can then later be merged with other snapshots of the same document, that have been changed by different people.
When you make a new chat, that changes the Automerge document, and it is snapshoted and saved to your browser storage. Every 5 seconds after a change, that snapshot is also sent to your PDS.
You could be modifying that document on multiple devices at the same time, though, and we need to make sure sure that we don't accidentally overwrite a snapshot from another device and lose some of our changes.
This is where we use a handy storage strategy that saves each snapshot with a unique name, based on its hash. If you are editing the document on two devices, and they both save snapshots, they will both get saved to your PDS because they have different hashes.
When you later open the app and need to download the latest version of your chat from your PDS, it will download all of the snapshots and simply merge them together, making sure you never lose data from any snapshot.
After you have loaded the snapshots, and then created a new snapshot, with the loaded data, you can safely delete the old snapshots that you loaded, because you know the new one has all the old changes merged into it.
This means that even without a transactional database, we can make sure we have solid, eventually consistent data synced to your PDS, even with multiple devices editing and syncing the same time.
How Do We Sync With Two People?
Now let's consider what happens in a direct message. If we want to load a direct message, we may have Automerge snapshots stored on our PDS with our changes, and more snapshots stored on our friend's PDS with their changes.
We are only allowed to edit our own PDS, so how do we chat? This is actually simple, when we load the chatroom, all we have to do is load the snapshots from our PDS and their PDS, and then we merge them together.
In this way all we have to do is both keep writing our snapshots to our own PDS and we can keep syncing things back and forth.
Encryption
Now a big caveat with PDSes is that essentially all of the data is public, and there's no way to make it private yet. This is something the ATProto devs want to improve on in the future. For now, though, we're stuck with everything being public.
So, for direct messages we introduce encryption. This isn't perfect, because while encryption is hard to break, it also gets easier over time as we make more powerful computers. So even if it's impossible to decrypt now, somebody might download it and keep it stored for years until they have a powerful enough computer or algorithm to crack the encryption.
Anyway, for now, it's the best we have, and in the future, we might have a paid or self-hosted sync service that you could use to keep your data non-public and encrypted.
OK, so we encrypt the data, but how? If you encrypt something you need a key of some sort. Where does the key come from?
Hosted Services
Because we need keys for encryption and signing, we introduce the first service that we are hosting ourselves: the Keyserver.
Keyserver
The keyserver is a tiny microservice that generates keypairs for each ATProto user that logs into Roomy.
This service technically has the secret keys for all users who log into Roomy by default. The reason for this is that if you lose your key, you can't decrypt any of your data. Your key has to be kept somewhere safe, and if we gave the user the key, they would have to make sure that they never lose it.
What happens if you used your phone, and you lost it? All your data would be inaccessible. People are used to being able to just say "I forgot my password", and they can get everything back. But if we let the user keep their key, we couldn't recover it for them if they lost it.
So, for starters, we keep your key safe for you, and that means that technically anybody with admin access to our keyserver can decrypt your messages.
The good news is that we will give you the option to manage your own key if you want to! That way you don't have to trust us if you want to accept the extra responsibility. In the future we will also add a fancy middleground option that works something like 2FA For Keypairs and will help us keep your keypair safer and more recoverable, but without giving us full access to it.
Also we specifically made the keyserver implementation absolutely tiny, so that it's easy to audit, and it's a completely independent service so no other code has access to its database.
We currently run the keyserver on the Deno Deploy serverless platform.
Roomy Router
At this point we are able to sync our messages back and forth across our PDSes by downloading snapshots. This cool part about this is that it lets us get the latest updates to the chatroom even if the other user is not online, since their PDS is always available to download from. But so far we don't have any instant messaging. That is why we introduce the router!
The Roomy router is the second service that we host ourselves, and its job is to get everybody connected to each-other for instant messaging.
All clients connect to the router with a WebSocket connection, and the router keeps track of which Automerge documents each client is interested in. It acts as a combination of a matchmaking server and a proxy, forwarding the clients' messages to each-other.
In the future the Router will also act as a WebRTC signaling server to allow clients to connect to each-other peer-to-peer and save bandwidth on the routing server.
The routing server does not store any data. It simply stays online, keeps track of who is connected, and tells clients when each-other join so that they can sync with each-other.
Once they are connected, clients can use the Automerge synchronization protocol to efficiently send each-other changes made to the chatroom without having to bother sending complete snapshots. Instant messaging achieved!
We currently run this service on a virtual machine in Digital Ocean, since it needs to be on all the time and is therefore unable to run on a serverless platform like Deno Deploy.
The router is one of the larger points of centralization in Roomy, and we want to explore ways of reducing the dependency on this central service. We'll go deeper into that below in the Decentralize it More section.
Space Coordinator
Now that we have realtime sync and the ability to load snapshots when other users are offline, we've covered all of the needs we have for direct messages. Public groups, or spaces, as we call them, are a little more complicated.
For instance, consider the scenario where we have 100 members in our group, which means that there are 100 different PDSes with chat snapshots on them. If we were the only user online and we wanted to get the latest updates for the chatroom, we would have to download at least 99 snapshots, from all of the other users, to make sure that we were up-to-date! That is way too many network requests, especially when you consider that most of those snapshots are going to contain data that we already have.
Also, how do we even know which users have joined the space and have snapshots to download?
Enter the Roomy space coordinator. The space coordinator serves the important purpose of keeping track of which users you need to download snapshots from to get the latest data.
Whenever a client updates its PDS with a newer snapshot for a space, it will send the space coordinator the list of hashes for all of the changes in that snapshot. The space coordinator can use those hashes to determine whether that snapshot contains the newest data, or if there are other PDSes that have newer data.
In this way the coordinator always knows which PDS has the latest data, and it also knows when it might be necessary to download snapshots from multiple PDSes. This is an edge case that might happen when somebody publishes their changes to their PDS, without downloading the updates from the latest snapshot.
After asking the space coordinator who to get the latest snapshot from, the client downloads the snapshot, merges it, and then updates its PDS records, potentially becoming the new user with the latest snapshot.
Finally the client connects to the router so that it can get instant notifications when other people join and make changes, and it doesn't need to ask the space coordinator for anything again as long as it's connected to the router.
The space coordinator is also a tiny service running on Deno Deploy.
Handle Resolver
Finally we have the handle resolver. It's just a mechanism for our web app to resolve user handles to their ATProto DID. Resolving a handle may require making DNS queries that are not allowed by web browsers, so the handle resolver service will do it for the app.
This microsocopic service is only 30 lines of code and it runs on Deno Deploy.
Frontend Server
That leaves only one remaining service: the frontend webserver. The Roomy frontend is just a static HTML, CSS, and JS web app. There's no server-side rendering, and it can be hosted almost anywhere.
Currently we are hosting it on Cloudflare pages because GitHub pages doesn't support SPA style routing. We should be able to make frontend builds with hash-style routing enabled so that you could host the frontend on GitHub pages if you wanted to.
Hosted Services Summary
In summary we have 4 microservices in addition to the frontend webserver.
So far, Roomy is actually extremely cost-effective for us to host! Unlike many ATProto apps that require running an AppView with its own database that replicates data from the firehose relay, Roomy operates with minimal centralized data.
We are not actually sure how many users we can handle with our current architecture. We've got to test things to find out what we can manage! There lots of optimizations we know we can make related to how we organize and manage the Automerge data, and we'll continue to improve on performance, but we won't know what our limits are until we hit them.
We're hoping we will be able facilitate very large numbers of people using Roomy for public discourse, without it representing a significant financial burden. We will still need to monetize Roomy to fund development, possibly by running bridges to other chat software, or providing sync servers for storing private data, but having lightweight infrastructure will make our funding needs much easier to meet. It also makes it far more feasible for other organizations to replace us if we are unable to continue hosting or development.
Decentralize it More!
Is Roomy decentralized? It's arguably at least as decentralized as ATProto, and probably a little more so because the router is a lighter-weight dependency than the ATProto relay.
But we can definitely try to make it more decentralized. Lets explore!
Keyserver
We can decentralize the keyserver by allowing you to add a record on your PDS that sets a custom public key. Then you can keep your kepair anywhere you want. You could also simply specify an alternative keyserver, so that you aren't using ours but get the same experience.
Keyservers are trivial to deploy and can be hosted on the Deno Deploy free plan and could be easily run on a VPS or adapted to run on all kinds of function-as-a-service platforms with different databases if desired.
Router & Space Coordinator
The router and the space coordinator are the most centralized services in Roomy. The router in particular because it has to connect to every single client that is using Roomy at any given point in time.
Things get very interesting, though, when we think about another feature that we want to add in the near future: resolving domains to Roomy spaces.
For example, right now we have a demo space running at:
https://roomy.chat/space/01JKVJAZFWJ65W2NJYCQQT1ERB
01JKVJAZFWJ65W2NJYCQQT1ERB is the space ID, but we'd like to have a nicer name than that. We want to be able to make our domain name, roomy.chat, resolve to our space's globally unique ID. That way our URL can be https://roomy.chat/space/roomy.chat.
If we are using DNS to resolve our space ID, then we could also use DNS to resolve custom router and space coordinator services.
If you were the organizer for a community, setting up your Roomy chat space, you could run your own router and space coordinator for just for your community. This would make you independent from our central services, but it would not allow you to compromise the data stored on your users' PDSes.
This starts to feel similar to some of the advantages of running Fediverse services like Mastodon for communies, but without the disadvantage of non-portable data and fragmented sign-in.
Handle Resolver & Frontend
The handle resolver is already essentially fully decentralized, all we would need to do is add an option for which resolver to use from the frontend.
The frontend also could be hosted anywhere, and could even be downloaded and opened from the local disk. Additionally, we'll have a desktop application in the future. The desktop application will be able to do handle resolution itself without a resolver, because desktop apps can query DNS.
The Desktop application may also allow you to chat over LAN / Bluetooth when you don't have internet.
plc.directory
plc.directory is the central service that powers identity on ATProto. It is the final major, centralized component that we depend on, if you don't count DNS and IP infrastructure in general.
This allows for things like pkdns to exist, which can provide alternative DNS resolution based on fully p2p records from the BitTorrent DHT.
The directory is responsible for providing a stable, universal identifier that is not a keypair, and that allows you to rotate your keys over time. Probably the biggest risk of plc.directory is that they can technically restrict your ability to make changes to your identity, such as rotating your keys.
They cannot forge your identity and make false changes to your ID records, but they can stop you from updating your records if they wanted to.
That said, it's a middleground solution that has some practical use, and it is what ATProto uses.
One possibility that could be interesting is if Roomy allowed you to use an alternative directory, instead of only the official one. That would have negative side-effects around connectivity to other ATProto services, and we would have to figure out how to namespace identifiers that could now come from multiple different directories.
That would probably involve having a different DID method for this purpose.
This is all speculative and we haven't done the proper research around what this would entail yet, but I think that there's a possibility here for making things even more decentralized.
It's also worth noting that ATProto already supports did:web IDs, but those have some serious disadvantages when it comes to ID portability, so it doesn't fix all of our problems. This is a great area to do more research on later.
Q&A Summary
Is Roomy P2P?
It's getting there. Eventually we can be nearly as peer-to-peer as browsers permit, and we will probably experiment more with different ways of bypassing the central services that are required today.
We want to give as many different ways for Roomy to survive with out us and other intermediaries as possible.
Is Roomy Local-First?
It's pretty close, and might be fully local-first eventually!
Right now we require you to login to ATproto before you do anything, and we use ATProto DIDs throughout the app to keep things simple. But we might be able to make a custom DID method that fits well with local-first so that you can get started without needing an ATProto identity.
Already Roomy does let you chat in a Room without internet access, once you've signed in.
Isn’t a CRDT excessive for a chat program?
I think you could build a chat app without a CRDT per-se, but there are a lot of things that it makes quite a bit easier for us to do.
- Synchronization: We don’t need to implement our own synchronization algorithm. We can just use the one from Automerge and not have to worry about how to make sure that everybody ends up with the same list of chats in the end.
- Serialization Format: Automerge comes with a compressed disk format that is quick to load and can be easily merged with other snapshots, which is important for our lock-free, concurrent storage on top of the remote PDS.
- Causality: Automerge documents are structured similar to git commits, which means that can tell all of the messages that were visible when any given message was posted. This could be useful for bringing clarity to the context in which a message made offline, and then later merged into the shared timeline, was posted. This could be useful for making a good UI for communication with extreme latency.
- Content Fluidity: We want the ability to transition (shape-shift) a thread to a channel or vice versa, and several other such moves or mergers of content which benefit a lot from having the git-like fine-grained edits offered by the CRDT which makes all these changes perfectly reversible and non-destructive.
- Digital Message Gardening: We want to play with the ‘chat interface’ as richly as demonstrated in Chatting with Glue, which is essentially our interactivity-benchmark.
So we’d probably end up just re-doing a lot of the things Automerge is doing for us. Also, while we do have to figure out exactly how to chunk out long chat histories still, I think there’s a good way to do it.
We’re not creating Automerge updates for every keystroke like you might in Google docs, so each chat message is its own commit with unique metadata, not altogether that different than what you’d need in any chat app.
We might need a little more metadata than normal, but it also compresses well.
Finally, one of the biggest reasons we are using Automerge in particular is due to the Ink & Switch lab’s cutting-edge research in Beelay & Beehive (soon to be renamed Keyhive).
- Capabilities: CRDT meshes perfectly with novel forms of distributed capabilities, like local-first access control.
Those combined will give us End-to-End Encrypted peer-to-peer groups and memory-efficient, fine-grained sync. Those features are really important to us for our long term goals, including growing into more use-cases than just chat.
Does Roomy Abuse the PDS Affordances?
Interesting question!
Roomy does something very different than most ATProto applications. It has a lot more traffic sent directly to the user PDSes, which is not how ATProto is generally intended to be used.
That said, I think that the PDS is one of the most crucially important pieces of the ATProto design, and the fact that users can have their own PDS gives us so much of the user agency that we desire.
It approaches the usefulness of the infeasible idea that "every user has their own Fediverse instance", but without many of the disadvantages.
Roomy's architecture isn't just useful for chat either! We are going to grow it and / or integrate it with public forum features, wiki pages, and digital gardening tools. This is an architecture that might be useful for all kinds of applications!
I'm honestly a little surprised we were able to combine this more p2p-leaning architecture with ATProto rather seamlessly so far.
Still, our overall "dream network" operates a bit more like a mesh of personal data stores that can sync with each-other, and may be optionally supplemented with features such as a firehose provided by a relay.
This remains a major experiment, but I think it's a useful direction that needs exploring.