Summary
On September 14th, 2026, a routine maintenance of the Furrylist database resulted in the Furrylist feeds being unexpectedly inaccessible for a period of over an hour. No malicious actor was involved and no data was lost.
Overview
Throughout the past year, Furrylist has stored 122 million likes made by the roughly 65 thousand manually vetted furry(-aligned) accounts, whose posts show up on the Furrylist feeds. We use these likes to calculate the popularity of posts for our Hot feeds.
These likes amass to around 70 GB of storage. We run Furrylist on a single, small server with only 150 GB of storage. A majority of likes isn't actually for furry posts. So, to save storage, we regularly delete irrelevant likes for posts that aren't made by furry accounts.
Why do we even store all these irrelevant likes? When you like a post, we need to store the like, even if the post doesn't exist in our database, due to the nature of Bluesky being a decentralized service. Due to a fun concept called eventual consistency, we may not learn about things happening on Bluesky in chronological order. In fact, we could know about a user liking a post before we learn about the existence and content of the post.
Therefore, we had been performing a routine maintenance during the past few days, where we cleaned up millions of irrelevant likes per day. This process is intentionally slow, so it doesn't impact the experience of using the feeds on Bluesky.
This process is complicated by the fact that “deleting all likes that don't have an associated post” is a very compute-intensive task because checking for the absence of data is a harder challenge than checking for the presence of data.
This maintenance window, we were particularly motivated to delete irrelevant likes because we wanted to further test a new algorithmic feed that we had been designing on-and-off for the past few months.
For this reason, we decided to speed up the cleanup process on September 14th. Instead of incrementally deleting the remaining likes, we would use a different process we had used before: copying all relevant likes to a separate table, clearing the main likes table, and restoring it with only the relevant likes. This method is significantly faster but also risks impacting the availability of the feeds.
In the first 10 minutes, we successfully created a temporary table that only contained around a tenth of the original likes. We started clearing the original likes table but during that accidentally deleted a database index that we didn't intend to delete, which caused the feeds to time out and no longer show any posts.
Although we could have recreated the index and have all feeds working again in about five to ten minutes, we decided to take the feeds service offline because it would make it faster to restore the relevant likes. If the feeds were offline, we wouldn't have to consider any impact to people using the feeds.
This second deviation from our plan caused us to make more mistakes. For example, restoring the likes should only take around three minutes but because we didn't turn the likes table into an unlogged table1 first, the process would've taken over 30 minutes instead.
We had also not deleted all indexes, which made restoring the main likes slower than necessary. Indexes speed up reading from a database table but make every write operation slower because indexes have to be updated.
After deleting all indexes and constraints on the main likes table and disabling said write-ahead logging, we were able to copy the likes into it from the temporary table. Now we were able to use concurrent index recreation2 to make the likes tables searchable again.
Finally, we reenabled the feed service. Due to being offline, it hadn't synced with the rest of the Bluesky network for over an hour. Thankfully, our Jetstream ingester can pick up where it left off and caught out with the Bluesky network in under 10 minutes, which fully resolved the outage.
Timeline
Time to recovery (user impact period): 1h 22m
Start of like deletion (Sept 14th): 16:24 UTC
Start of new deletion method: ~20:30 UTC
Accidental index deletion: ~20:41 UTC
Feeds are unreachable: ~20:53 UTC
Incident communicated (post 1): 21:36 UTC
Data restored and indexes recreated: 21:59 UTC
Incident update (post 2): 22:07 UTC
Incident resolved (post 3): 22:15 UTC
Five whys
The five whys is an analytical method to understand the cause-and-effect chain that led to an incident.
Furrylist had a full feed outage on September 14th for approximately 1h 22m.
- 1.
Why? We turned off the feed and ingest services.
- 2.
Why? They unexpectedly became unresponsive during a planned maintenance.
- 3.
Why? We accidentally deleted an index that is necessary for production use.
- 4.
Why? We were trying to bulk-delete irrelevant likes using a faster, riskier process.
- 5.
Why? We were impatient to delete all irrelevant likes.
Fundamentally, the outage and user impact was caused by a combination of three factors:
- 1.
We made a judgement error that a fast data was more valuable than a safe data cleanup.
- 2.
We accidentally deleted an index, which we could have just recreated and reverted to the slow and safe cleanup process.
- 3.
The user impact was exacerbated by doing the data cleanup during the busiest time of the day.
Corrective actions and lessons learned
We already addressed the major scalability concern of having to regularly clean up irrelevant likes in a computationally expensive process by no longer storing a majority of these irrelevant likes. Even if the post doesn't exist yet, as long as we know the liked post was made by a furry account, we can track it. That will saves us around 75% of disk space and speed up our database queries. It will also make future manual cleanups of this scale unnecessary.
We will still have to semi-regularly delete likes that are older than our retention window of one year. But since we don't have to cross-reference the existence of posts with likes, this operation is a classic database deletion based on the indexed created_at column, which can happen at any time during regular operations.
Although Furrylist is a volunteer-run project with no expectation of high availability, tens of thousands of (animal) people use Furrylist feeds every day as part of curating their own digital third places. We shouldn't interfere with this for silly reasons like our own impatience.
Additionally, we'll be more careful in the future when using destructive actions on our production database. Likewise, we should not have deviated from the original plan, especially because the incident ended up wasting more of our time than letting the original plan finish.