Designing a High-Concurrency Real-Time Barrage System for a Million Concurrent Viewers

1. Why Barrage Systems Are Hard

At first glance a barrage is just "sending messages": a user types a line, and the server pushes it to everyone else in the room. But as room occupancy grows from a few hundred to a million, this seemingly simple module quickly becomes the most fragile part of the entire live-streaming system.

The reason is its fan-out characteristic. Suppose a room has 800,000 concurrent viewers and peak barrage volume reaches 120,000 messages per second. If every barrage were pushed verbatim to every person, the theoretical push volume would be astronomical — any monolithic architecture would saturate CPU and bandwidth within minutes, with latency degrading from a few hundred milliseconds to several seconds, surfacing to users as "barrages won't load, the screen is lagging."

So a barrage system is far more than "pushing messages." It requires careful trade-offs among real-time delivery, delivery rate, and cost. This article dissects the full design — from monolithic direct push to tiered fan-out — in the order the architecture naturally evolves, along with the engineering details that are easy to overlook.

2. Analyzing the Problem Space

2.1 What Makes a Barrage System Special

Barrage looks a lot like ordinary IM chat, but it has a few fundamental differences:

• Extremely skewed read/write ratio: one person sends a barrage, and hundreds of thousands of people in the room have to receive it. Few writes, many reads — a textbook one-write-many-read fan-out model.

• High real-time requirements: if barrage latency exceeds 3 seconds, users feel "the host is ignoring me" and the experience falls off a cliff.

• Discardability: unlike payment messages, dropping a few barrages is essentially unnoticeable to users. This property is crucial — it's the foundation of every degradation strategy later.

• Extreme hot-spot concentration: a room is normally a few hundred people, but a major stream can gather a million in an instant. Traffic is heavily skewed.

2.2 Core Scenarios

Scenario 1: Ordinary rooms (hundreds to tens of thousands)

Host room, occupancy: 8,000
├─ User sends barrage "host is awesome"
├─ Server receives → validate → store
└─ Fan out to all 8,000 connections in the room

This is the most common case. 8,000 pushes are easily handled on a single machine.

Scenario 2: Hot rooms (hundreds of thousands to a million)

Large live room, occupancy: 800,000
├─ Peak barrage volume: 120,000 msgs/sec
├─ Each barrage must fan out: 800,000 pushes
└─ Theoretical push volume: 120,000 × 800,000 = 96 billion/sec (impossible)

This is the most challenging scenario. If every barrage were pushed verbatim to every person, the push volume is astronomical — no cluster can bear it. Here we must introduce merging, sampling, and rate-limiting.

Scenario 3: Platform-wide broadcast (cross-room)

User sends a "Super Rocket" in room A
↓
Triggers a site-wide banner; every online user (including other rooms)
sees across the top: "User X sent a Super Rocket in [some room]"

These messages are low in volume but universal in reach. They need a dedicated broadcast channel and must not be mixed with in-room barrage.

2.3 Target Metrics

• Barrage end-to-end latency P99: < 1 second

• Concurrent online per room: 1,000,000+

• Delivery rate (ordinary barrage): > 90% (degradation drops allowed)

• Delivery rate (paid barrage / gifts): > 99.9% (never dropped)

• WebSocket long connections per machine: 100,000+

3. Architecture Evolution

3.1 V1: Monolithic WebSocket Direct Push

The original architecture was dead simple: one gateway server, users connected directly, and sending a barrage meant iterating over every connection in the room and pushing to each.

         ┌──────────────────┐
User ───→ │  WebSocket GW     │ ───→ iterate room conn list
         │  (one/few boxes)  │      conn.Send() one by one
         └──────────────────┘

Where it broke:

• Single-machine connection ceiling around 50,000 — 800,000 people simply couldn't connect.

• Pushing was a synchronous loop; one hot barrage meant looping hundreds of thousands of times, pegging the CPU instantly.

• One gateway dies and every user on it drops.

This version worked for small rooms and collapsed the moment a big stream arrived.

3.2 V2: Gateway Cluster + Message Queue Decoupling

The first optimization step. The core idea is to separate the access layer from the logic layer and use a message queue to shave peaks.

                  ┌─────────────┐
User ──long conn──→ │ Access GW    │ (many boxes, connections only)
                  │ Gateway×N   │
                  └──────┬──────┘
                         │ upstream barrage
                         ↓
                  ┌─────────────┐
                  │   Kafka     │ peak buffering
                  └──────┬──────┘
                         ↓
                  ┌─────────────┐
                  │ Logic layer │ validate/store/filter
                  │ Logic×N     │
                  └──────┬──────┘
                         │ downstream push commands
                         ↓
                  ┌─────────────┐
                  │ push to GW  │ → fan out to users
                  └─────────────┘

Key design points:

• Access gateways only maintain long connections and move bytes — no business logic — so they scale horizontally.

• Connection info (which user is on which gateway in which room) is registered in Redis.

• Upstream barrages go to Kafka first; the logic layer pulls at its own consumption rate, naturally shaving peaks.

• After processing, the logic layer sends push commands back to the right gateways via internal RPC.

This version solved the single-machine bottleneck, but during big streams the fan-out push was still a disaster — in a million-person room, one barrage still meant a million pushes across multiple gateways.

3.3 V3: Layered Fan-Out + Merging and Rate-Limiting (final form)

This step is what truly solves the hot-spot problem. The core is a three-part approach: room sharding, message merging, and tiered rate-limiting.

                    ┌──────────────────────┐
   upstream ──────→  │ Logic: barrage proc.  │
                    │  - profanity filter    │
                    │  - rate limit (per user)│
                    │  - priority tagging    │
                    └──────────┬───────────┘
                               ↓
                    ┌──────────────────────┐
                    │ Merge buffer (100ms)  │
                    │ batch high-freq barrage│
                    └──────────┬───────────┘
                               ↓
            ┌──────────────────┼──────────────────┐
            ↓                  ↓                  ↓
      ┌──────────┐      ┌──────────┐      ┌──────────┐
      │ GW shard1 │      │ GW shard2 │      │ GW shardN │
      │ 100k conns│      │ 100k conns│      │ 100k conns│
      └──────────┘      └──────────┘      └──────────┘

   Big rooms split across gateways by connection count. The logic layer
   sends each batch to each gateway once; the gateway fans out locally to
   the connections it owns (in-memory, no network hop — extremely fast).

Three core optimizations:

1. Local fan-out: instead of pushing to each user directly, the logic layer sends "this room's batch of barrages" once to each gateway holding connections for that room, and the gateway fans out in local memory. Cross-network pushes drop from a million to a few dozen (the number of gateways).

2. Message merging: barrages within a 100ms window are packed into one batch and sent down together, rendered by the front end in a single pass. The higher the barrage volume, the bigger the merging payoff.

3. Tiered rate-limiting: ordinary barrages are proportionally sampled and dropped during peaks, while paid/gift barrages travel a high-priority channel and are never dropped.

4. Core Implementation

4.1 Long-Connection Management

Holding 100,000+ connections per machine comes down to an epoll + goroutine model, avoiding one thread per connection. Using Go as an example:

// Connection manager: each room holds a set of connections
type Room struct {
    ID    string
    conns sync.Map // connID -> *Connection
    count int64    // atomic counter
}

type Connection struct {
    ID     string
    UserID int64
    ws     *websocket.Conn
    sendCh chan []byte // one send buffer channel per connection
}

// A dedicated write goroutine per connection, to avoid concurrent
// writes to the same ws (which would panic)
func (c *Connection) writeLoop() {
    for msg := range c.sendCh {
        // write with a deadline so a slow connection can't drag everything down
        c.ws.SetWriteDeadline(time.Now().Add(2 * time.Second))
        if err := c.ws.WriteMessage(websocket.BinaryMessage, msg); err != nil {
            c.close() // close immediately on write failure
            return
        }
    }
}

// Push: non-blocking; if the buffer is full, drop (barrage is discardable)
func (c *Connection) push(msg []byte) {
    select {
    case c.sendCh <- msg:
    default:
        // a full buffer means this user's network is slow; drop this
        // message rather than blocking the fan-out
        atomic.AddInt64(&droppedCount, 1)
    }
}

Pitfall to avoid: with a blocking sendCh <- msg, once a user with a bad network fills their buffer, the entire room's fan-out goroutine stalls on them and the whole room's barrage freezes. Switching to non-blocking send plus drop-when-full keeps a slow user from affecting anyone but themselves.

4.2 In-Room Local Fan-Out

// The gateway receives a batch from the logic layer and fans out locally
func (r *Room) broadcast(batch []byte) {
    r.conns.Range(func(_, v interface{}) bool {
        conn := v.(*Connection)
        conn.push(batch) // non-blocking push
        return true
    })
}

Fan-out is a pure in-memory operation. One pass over a 100,000-connection room takes roughly 10–20ms, which is perfectly acceptable.

4.3 Message Merge Buffer

// Merger: aggregates barrages per room within a 100ms window
type Merger struct {
    buffer map[string][]*Barrage // roomID -> barrage list
    mu     sync.Mutex
}

func (m *Merger) add(b *Barrage) {
    m.mu.Lock()
    m.buffer[b.RoomID] = append(m.buffer[b.RoomID], b)
    m.mu.Unlock()
}

// Flush every 100ms, pack and send down
func (m *Merger) flushLoop() {
    ticker := time.NewTicker(100 * time.Millisecond)
    for range ticker.C {
        m.mu.Lock()
        batches := m.buffer
        m.buffer = make(map[string][]*Barrage)
        m.mu.Unlock()

        for roomID, list := range batches {
            // sample when a single room is over quota, keeping only some ordinary barrages
            list = sample(roomID, list)
            packed := encodeBatch(list) // one encode, many barrages
            dispatchToGateways(roomID, packed)
        }
    }
}

What merging achieves: for a stream peaking at 120,000 barrages/sec, merging can reduce downstream batches to about 2,000/sec (averaging 60 barrages each) — a 60× drop in downstream push frequency. The front end renders received batches across frames with requestAnimationFrame, so there's no jank.

4.4 Tiered Rate-Limiting and Sampling

Barrage's "discardable" property is pushed to its limit here:

func sample(roomID string, list []*Barrage) []*Barrage {
    online := getRoomOnline(roomID)
    // dynamically compute the per-batch barrage cap for this room by occupancy
    limit := calcLimit(online) // e.g. a million-person room shows at most 50 per batch

    var kept []*Barrage
    var normal []*Barrage
    for _, b := range list {
        if b.Priority == HighPriority {
            kept = append(kept, b) // paid/gift barrages kept unconditionally
        } else {
            normal = append(normal, b)
        }
    }

    // sample ordinary barrages down to the remaining quota
    remain := limit - len(kept)
    if remain > 0 && len(normal) > remain {
        normal = randomSample(normal, remain)
    }
    return append(kept, normal...)
}

The product-level elegance: in a million-person room, even showing just 50 barrages per second feels seamless to users — the screen can't hold more anyway. But not a single paid barrage is dropped, which protects revenue. This strategy preserves both the experience and the core business data during traffic floods.

A quick note from the business side: paid barrages and gift messages have to be "never dropped and shown first" precisely because they tie directly to streamer and platform revenue — and to the sense of recognition users are paying for. That makes the visual payoff on this high-priority channel just as worth investing in: a well-crafted animated gift flying across the screen tends to drive far more conversion and engagement than plain text. This is exactly the niche XingJi Shop focuses on, providing ready-to-integrate animated virtual gifts and effect assets for voice and live-streaming apps. Engineering and content are really two sides of one coin: the backend guarantees the gift barrage arrives, undropped and low-latency, while great animation makes that barrage worth every cent.

4.5 Connection Routing and State Management

With a million connections spread across dozens of gateways, how does the logic layer know which gateways to send a room's batch to?

Redis maintains a two-level mapping:

room:{roomID}:gateways → Set{gw-01, gw-07, gw-15, ...}
  (which gateways hold this room's connections)

gw:{gwID}:rooms → Set{room-A, room-B, ...}
  (which rooms this gateway carries)

Before dispatching, the logic layer queries room:{roomID}:gateways to get the gateway list and sends only to those, rather than broadcasting to all gateways. As gateways come and go, these two sets are updated, with Redis expiry plus heartbeat renewal ensuring eventual consistency.

Pitfall to avoid: without this routing layer, the logic layer broadcasts each room's batch to every gateway, which then discards it if it doesn't hold that room. This saturates internal gateway-cluster bandwidth with useless messages. Adding precise routing typically cuts internal traffic by around 90%.

4.6 Upstream Rate-Limiting

To stop a single user or script from spamming:

// Redis-based sliding-window rate limit: max 2 barrages per second per user
func allowSend(userID int64) bool {
    key := fmt.Sprintf("rate:barrage:%d", userID)
    cnt, _ := redis.Incr(key)
    if cnt == 1 {
        redis.Expire(key, time.Second)
    }
    return cnt <= 2
}

Over-limit barrages are dropped right at the access layer — they never even reach Kafka — blocking spam traffic at the source.

5. Common Engineering Issues and Mitigations

5.1 Thundering Herd: the connection storm when the host enters

The instant a big host goes live, hundreds of thousands of fans' clients receive the push and rush in to establish WebSocket connections simultaneously, overwhelming the gateways with connection requests.

Solution: add random jitter on the client (a 0–3 second random delay before connecting) to flatten the spike; on the gateway side, token-bucket rate-limit new connections, queuing and retrying anything over the rate.

5.2 Out-of-Order Messages: barrage order scrambled after merging

After passing through multiple gateways and goroutines, merged batches can arrive in an order that doesn't match the send order. Scrambled ordering of gift barrages in particular directly hurts the paid experience.

Solution: every barrage carries a server-generated incrementing sequence number (per room); the front end sorts by sequence number before rendering. Order is preserved inside each merged batch.

5.3 Memory Leak: dropped connections not cleaned up

During load testing, gateway memory often keeps climbing. The root cause is typically users dropping abnormally (no proper close frame), where the server's connection objects and Redis routing aren't cleaned up promptly, leaving piles of zombie connections in sync.Map.

Solution: add heartbeat detection per connection (no pong for 30 seconds means offline); on going offline, immediately clean up the in-memory object, the Redis routing, and the send channel, which keeps memory stable.

5.4 Hot Rooms Crushing a Single Gateway

If one gateway happens to carry several big rooms, its connection count can far exceed the others and overload it.

Solution: introduce connection-count-aware load balancing — new connections go first to the gateway with the lowest current load; big rooms are forced to spread across multiple gateways, with a cap on how many connections for the same room a single gateway may hold.

5.5 Site-Wide Broadcast Swamping the Room Channel

If site-wide banners (rocket broadcasts) and in-room barrage share the same downstream channel, a single site-wide broadcast can hog every gateway's downstream queue and delay in-room barrage across the board.

Solution: broadcast messages travel a dedicated channel and dedicated goroutine pool, physically isolated from in-room barrage so the two never interfere.

6. Performance Comparison

As the architecture evolves from monolith to layered fan-out, per-room capacity grows roughly 20x while peak latency compresses from seconds to sub-second. These gains come not from throwing hardware at the problem, but from fully exploiting barrage's characteristics: discardable, read-heavy, and hot-spot concentrated.

7. A Few Reflections

A core insight in barrage system design: the heart of it isn't "how do I push every message out," it's "how do I gracefully push a little less when I can't keep up."

Unlike strongly-consistent systems such as payments and orders, barrage's "discardable" property is a weapon. Once you internalize that users can't actually read every barrage anyway, you can reach for sampling, merging, and rate-limiting boldly. The choice of technical approach is often a direct mapping of how well the business characteristics are understood.

Hot spots, too, are the ultimate test of nearly every real-time system. An architecture that runs fine day to day shows its true colors the moment traffic skews. Layered fan-out, local fan-out, precise routing — at bottom they're all a fight against hot spots.

Real high concurrency isn't making a system handle every request; it's making a system refuse to collapse under any amount of traffic.

Hopefully this offers a useful reference for engineers designing real-time push systems.