@kido/db-shardingCopy Anchor Link

Architecture Paradigm: MurmurHash3 Single-Shard Routing + 150-Virtual-Node Consistent Hash Ring with RF=3 Clockwise Replica Walk + Discord Anti-Hotspot Time Bucketing + 2-Level Native PostgreSQL Sub-Partitioning + Little's Law Queueing Headroom Guard ($\rho < 0.85$).

💥 The Real-World BottleneckCopy Anchor Link

When relational databases grow past hundreds of gigabytes, monolithic single-table architectures collapse:

1. B-Tree Index Buffer Pool Eviction: As tables exceed 50 million rows, the B-Tree primary index no longer fits into RAM buffer pool memory. Every index lookup triggers random disk I/O reads, inflating query latency from 1ms to >800ms.
2. The "100% Capacity" Queueing Trap (Little's Law Disaster): Teams frequently size replica clusters for 100% capacity ($\rho = \lambda / \mu = 1.0$). According to queueing theory ($W = W_s / (1 - \rho)$), when utilization reaches \$1.0$, latency explodes asymptotically from 20ms to 1,420ms, causing cascading connection starvation.
3. The Dangerous Modulo Rebalancing Cascade: Naive sharding (hash(id) % N) forces 80% of total data to move when adding a 5th node to a 4-node cluster (\$1 - 1/N$). The resulting rebalancing network storm downs production databases for hours.
4. Historical Deletes Starving Autovacuum: Running DELETE FROM logs WHERE created_at < NOW() - INTERVAL '90 days' generates millions of dead tuples and exclusive table locks, freezing all customer mutations.

🏛️ 3-Tier Production Evolution LadderCopy Anchor Link

📦 Architecture & Core API ExportsCopy Anchor Link

@kido/db-sharding adheres strictly to Designing Data-Intensive Applications (DDIA Chapter 6):

1. Modulo Hash & MurmurHash3 Router (src/murmurhash.ts, src/hash-router.ts)Copy Anchor Link

• murmurhash3_32(key, seed) & murmurhash3_64(key): Ultra-fast 32-bit and 64-bit non-cryptographic hash functions with optimal avalanche characteristics.
• HashShardRouter: Deterministic single-shard router resolving target shard nodes. Supports scatter-gather parallel execution across all shards for non-sharded secondary queries.

2. Consistent Hash Ring with Virtual Nodes (src/consistent-hash-ring.ts)Copy Anchor Link

• ConsistentHashRing: 150-virtual-node consistent hash ring distributing token keys uniformly across the \2^32 integer space with standard deviation $< 3%$.
• Adding a 5th node migrates only $\approx 20%$ of keys (\$1/N$), preventing rebalancing cascade failures.
• buildDiscordBucketKey(entityId, timestamp, bucketMinutes): Generates time-bucketed compound keys preventing single viral users from overloading physical shard nodes.
• getReplicaNodes(key, replicationFactor): Implements Cassandra/DynamoDB-style clockwise ring walking to place replicas across distinct physical hardware nodes.

3. Native 2-Level Compound Sub-Partitioning (src/compound-partition.ts)Copy Anchor Link

• CompoundPartitionManager: Automated DDL generator for PostgreSQL 2-Level Declarative Sub-Partitioning:
  • Level 1: PARTITION BY HASH (user_id) into $K$ shards.
  • Level 2: PARTITION BY RANGE (created_at) into monthly tables.
• withCompoundShardPruning(userId, date): Enforces query constraints so the PostgreSQL query planner prunes $(K-1)$ hash shards and all past months, reading exactly 1 physical leaf table.
• buildCompoundDetachSQL(tableName, partitionName): Generates zero-lock ALTER TABLE ... DETACH PARTITION ... CONCURRENTLY statements for historical archival.

4. Read-Write Splitter & Little's Law Guard (src/tier0-sme-read-router.ts)Copy Anchor Link

• SmeReadWriteSplitter: Intelligent query router directing writes to Primary and load-balancing reads across Replicas.
• calculateLittleLawLatency(input): Validates cluster headroom ensuring queue utilization $\rho \le 0.85$.
• Built-in sticky session awareness guaranteeing Read-Your-Own-Writes consistency for users who recently submitted a mutation.

💻 Production Implementation & Code SnippetCopy Anchor Link

Read-Write Splitter & Consistent Hash RouterCopy Anchor Link

import { 
  SmeReadWriteSplitter, 
  ConsistentHashRing,
  buildDiscordBucketKey 
} from '@kido/db-sharding';

// 1. Initialize Read-Write Splitter with Little's Law Headroom Guard
export const dbRouter = new SmeReadWriteSplitter({
  primary: { id: 'primary-db', host: '10.0.0.1', isHealthy: true },
  replicas: [
    { id: 'replica-01', host: '10.0.0.2', isHealthy: true, weight: 1 },
    { id: 'replica-02', host: '10.0.0.3', isHealthy: true, weight: 1 },
    { id: 'replica-03', host: '10.0.0.4', isHealthy: true, weight: 1 },
  ],
  stickySessionTtlMs: 3000, // Read-your-own-writes sticky window
  maxReplicaUtilization: 0.85, // Headroom guard (rho <= 0.85)
});

// 2. Initialize Distributed Consistent Hash Ring (Tier 3)
export const hashRing = new ConsistentHashRing({
  virtualNodes: 150,
});

hashRing.addNode('shard-us-east-1');
hashRing.addNode('shard-us-west-1');
hashRing.addNode('shard-eu-central-1');

// 3. User Message Ingestion Handler (Discord Pattern)
export async function insertMessage(guildId: string, authorId: string, content: string) {
  // A. Protect against hot-spot writes using Discord time-bucketing
  const bucketKey = buildDiscordBucketKey(guildId, new Date(), 60); // 1-hour bucket
  
  // B. Determine target shard node via consistent hash ring
  const targetShard = hashRing.getNode(bucketKey);
  const replicaShards = hashRing.getReplicaNodes(bucketKey, 3); // RF=3

  // C. Execute write on the selected shard's primary
  console.log(`Writing message to Primary Shard: ${targetShard.id}, Replicating to:`, replicaShards.map(r => r.id));

  // D. Mark user session as dirty for Read-Your-Own-Writes consistency
  dbRouter.markSessionDirty(authorId);

  return { success: true, targetShard: targetShard.id };
}

⚡ Benchmark & Verified k6 PerformanceCopy Anchor Link

Benchmarked against single-table monolithic PostgreSQL vs 2-Level Sub-Partitioning at 100M rows:

🚀 Surgical Extraction & CLI InstallationCopy Anchor Link

# Install directly via Kido CLI
npx @kido/cli add @kido/db-sharding