How Discord Stores Billions of Messages

Discord handles an unfathomable amount of chat data. This post breaks down their architectural evolution, largely inspired by case studies from ByteByteGo and Reddit engineering communities.

The Early Days: MongoDB

In 2015, Discord started with MongoDB. It was fast to iterate with, but as they hit millions of messages a day, index sizes exceeded RAM. MongoDB's B-Tree indexing meant that random reads on historical messages required costly disk I/O, causing massive latency spikes.

The Migration to Cassandra (2017)

To solve the scaling issue, they moved to Apache Cassandra. Cassandra uses an LSM-Tree (Log-Structured Merge Tree) and is optimized for heavy write workloads.

Partition Key: `channel_id` (Messages are grouped by channel).

Clustering Key: `message_id` (Snowflake ID sorted chronologically).

This schema was brilliant. Because users read messages sequentially in a channel, Cassandra could fetch a contiguous block of messages straight from disk.

#

The "Hot Partition" Problem

A server with 100,000 members going wild creates a "hot partition." One node in the Cassandra ring gets flooded with writes, starving others. Discord mitigated this by batching writes and shedding load, but GC (Garbage Collection) pauses in Java-based Cassandra remained a persistent issue.

The Final Evolution: ScyllaDB (2022)

Cassandra's Java garbage collection caused unpredictable tail latency (P99 spikes). Discord eventually migrated to ScyllaDB—a Cassandra clone written in C++. ScyllaDB uses a thread-per-core architecture, entirely bypassing Java's GC issues. This reduced tail latency to low milliseconds, successfully allowing Discord to store trillions of messages with rock-solid stability.