We said we'd fix it. Here's where we are.

Last week we wrote about the Bluesky DDoS outage and what it did to Chunky Guys. Short version: Bluesky went down and we went down with it; but that can't keep happening. We said we were going to start building our own infrastructure, so here's an update...

But first, it's worth explaining how the system on which we run actually works. Not because you need to know it to use the app, but because the rest of this post will make a lot more sense.

How ATProto works (the short version)

Bluesky runs on a protocol called ATProto. It's a protocol, not platform. Think of it like email: anyone can run their own mail server, but most people just use Gmail or Proton. The protocol is the common language that lets all the servers talk to each other.

Your data (your posts, your likes, who you follow) lives on something called a PDS, a Personal Data Server. That's your home base. When you do anything on the network, your PDS sends it out to what's called the firehose: one big live stream of everything happening across the entire network. Every post, every like, every follow, every profile change, all of it, flowing past in real time. Right now the firehose pushes somewhere between 500 and 1000 events per second.

Then there's the AppView. The AppView takes all that raw data and turns it into something you can actually use: your timeline, your notifications, search results. When you open Bluesky (or Chunky Guys), the app asks an AppView "what should I show to this person?" and the AppView answers.

Here's where it gets awkward: until now, there was only one AppView that mattered, and Bluesky runs it. When the DDoS hit, the firehose kept going, everyone's data was safe; but the AppView went down, and every app that needed it went down too. Including ours.

What we've built

We've started building a service that listens to the firehose directly and keeps its own copy of everything Chunky Guys needs. Not the whole network (that would be massive), just the bit that matters to us: posts by CG members, the replies and quotes they get, who's liking and reposting what, who's following who, and what everyone's profiles look like. The full picture of the community, updated in real time.

This isn't a demo, but it's not finished either. It's running against the live network right now, pulling from the real firehose, filtering in real time, and writing to our database. A week ago none of this existed. We're about halfway through the build and it's still in active development, but already we're capturing the data Chunky Guys needs to show you your timelines, profiles, threads, and notifications.

What's next

Having the data is half the job. The other half is wiring the app to read from our copy instead of asking Bluesky. That work hasn't started yet, and it's not small. Until it's done, the app still reads from Bluesky, so we're still tied to them for now.

We're not going to rush it. The whole point of this is reliability. Cutting corners to get there faster would miss the point entirely. But when it lands, the aim is clear: next time Bluesky's AppView has a bad day, Chunky Guys keeps going. Your feeds load. Your notifications arrive.

Fixing what the outage broke

The DDoS also showed up some real problems in how our own systems cope when Bluesky falls over. Some of you got stuck in a state where the app looked like you were logged in but nothing worked. The connection to Bluesky had died, but we weren't spotting it properly, so the app just kept trying and failing quietly instead of logging you out and letting you start clean.

We've dug into it and found a handful of related issues in how we handle sessions and how we deal with failures. None of it is exciting work. It's the kind of plumbing that nobody notices when it works and everyone notices when it doesn't. We're working through it.

Where this leaves us

We're a week in. The foundations are down, the data is flowing, and there's a clear path to the finish. It's not done and we're not going to pretend it is. But it's moving fast, and we'll keep sharing progress as we go.

If you've got questions, thoughts, or just want to tell us something, we're here. Talk to us.