01. YouTube-Scale Video View CounterCopy Anchor Link

Production Paradigm: Adaptive Slotted Write + HyperLogLog In-Memory Deduplication + Anti-F5 Fingerprinting + Write-Behind Coalescing.

💥 The Real-World BottleneckCopy Anchor Link

When a video goes viral (e.g., a breaking news announcement or music release), it can receive 100,000 views per second. If each view triggers a naive relational database update:

-- The Naive Disaster: 100,000 concurrent updates hitting a single row
UPDATE videos SET view_count = view_count + 1 WHERE id = 'dQw4w9WgXcQ';

This immediately triggers:

1. Row-level lock contention: Transactions queue up waiting for the row lock, inflating latency from 2ms to >15,000ms.
2. Connection pool exhaustion: PostgreSQL or MySQL max connections are exhausted within seconds.
3. Database crash: High CPU load from lock arbitration and replication storms.

Furthermore, spam bots and malicious users spamming F5 / reload should not distort analytics or drain server resources.

🏛️ High-Concurrency ArchitectureCopy Anchor Link

This blueprint solves the problem using an adaptive 3-tier write pipeline combined with an in-memory HyperLogLog filter and single-query micro-batch coalescing.

Key Architectural PhasesCopy Anchor Link

1. Anti-F5 Fingerprint & HyperLogLog Dedupe:
   • Every play request passes an IP and User-Agent hash through a zero-dependency HyperLogLog counter (HyperLogLogCounter).
   • Occupies less than 1 KB of RAM per video while guaranteeing an error rate under 3%.
2. Adaptive Slot Sizing (The Famous YouTube "301 Views" Freeze):
   • Videos under 300 views use simple row updates.
   • Moderately popular videos transition to a 16-slot sharded counter (SlottedCounter), distributing row lock contention across 16 tuples (slot_id = murmurhash(viewer_id) % 16).
   • Viral videos (>10,000 views) bypass relational updates entirely, routing to an in-memory Valkey HINCRBY buffer.
3. Single-Query Write-Behind Coalescer:
   • Every 1,000ms, a background worker flushes the Valkey buffer using a single parameterized SQL query:

   Copy Text

   UPDATE video_stats AS v
   SET views = v.views + batch.views
   FROM (SELECT * FROM UNNEST($1::text[], $2::int[])) AS batch(video_id, views)
   WHERE v.id = batch.video_id;

   • 100,000 writes are collapsed into 1 single disk write per second.

🧩 Composed Functional SlicesCopy Anchor Link

This blueprint is built from four self-contained slices:

💻 Code ImplementationCopy Anchor Link

mkdir -p src/lib/hot-counter
cp packages/hot-counter/src/youtube-view-engine.ts src/lib/hot-counter/
cp packages/hot-counter/src/valkey-write-behind.ts src/lib/hot-counter/
cp packages/hot-counter/src/slotted-counter.ts src/lib/hot-counter/

import { YouTubeHybridViewEngine, HyperLogLogCounter } from '@/lib/hot-counter/youtube-view-engine';
import { ValkeyWriteBehindCounter } from '@/lib/hot-counter/valkey-write-behind';
import { db } from '@/lib/db';

// 1. Initialize Deduplication & Write-Behind Coalescer
const hllDeduper = new HyperLogLogCounter();
const writeBehind = new ValkeyWriteBehindCounter({
  flushIntervalMs: 1000,
  maxBatchSize: 10000,
});

// 2. Initialize YouTube-Grade Adaptive Engine
export const videoViewEngine = new YouTubeHybridViewEngine({
  viralThreshold: 301,
  hllDeduplicator: hllDeduper,
  async onFlushBatch(counts) {
    // Single batch update replaces 100k individual database writes
    await db.$executeRaw`
      UPDATE video_stats AS v
      SET views = v.views + batch.views
      FROM (SELECT * FROM UNNEST(${counts.videoIds}, ${counts.increments})) AS batch(video_id, views)
      WHERE v.id = batch.video_id;
    `;
  },
});

// 3. API Handler
export async function recordVideoView(videoId: string, viewerIp: string, viewerId?: string) {
  const fingerprint = `${viewerIp}:${viewerId ?? 'anonymous'}`;
  const isUnique = videoViewEngine.recordView(videoId, fingerprint);
  return { success: true, counted: isUnique };
}

📊 Production Scale Benchmark & VerificationCopy Anchor Link

Verified against the GonnaCrash Level 24 (counter_at_scale) benchmark: