@kido/anti-spamCopy Anchor Link

Architecture Paradigm: Single-Pass Aho-Corasick Multi-Pattern Trie Automaton + 64-Bit Charikar SimHash Near-Duplicate Indexing + NFC/NFD Unicode & Homoglyph Normalization + Fail-Open AI Semantic Verification.

💥 The Real-World BottleneckCopy Anchor Link

Community platforms, commenting systems, marketplace reviews, and lead forms face intense automated abuse:

1. The O(N x M) Linear Search Event Loop Hang:

   Copy Text

   // The Naive Disaster: Linear search over thousands of blacklisted terms
   for (const word of blacklist) {
     if (content.toLowerCase().includes(word)) throw new Error("Spam detected");
   }

   Checking 2,000 blacklisted phrases against incoming long-form comments starves the Node.js/Bun event loop, inflating latency from 2ms to >150ms per request and causing HTTP 504 timeouts.
2. The "Scunthorpe Problem" (Severe False Positives): Naive substrings block legitimate users. Banning dm blocks admin and roadmap; banning sex blocks Sussex and Essex; banning bán (sell) in Vietnamese blocks cơ bản (fundamental) or bản sắc (identity).
3. Evasion Tactics (Zero-Width Characters & Homoglyphs): Spammers evade simple regex checks by inserting invisible zero-width spaces (\u200B, \uFEFF), decomposing Unicode diacritics, or mixing Cyrillic letters (а, е, о) into Latin words (b.á.n, z a l o, t.m.e).
4. Copy-Paste Bot Inundation (Cross-Post Duplication): Botnets flood forums by slightly modifying spam text (changing one punctuation mark or greeting). Standard hash checks (md5/sha256) treat every post as completely new, failing to stop cross-post spam.

🏛️ 3-Tier Production Evolution LadderCopy Anchor Link

📦 Architecture & Core API ExportsCopy Anchor Link

@kido/anti-spam is organized into clean, zero-dependency modules:

1. Normalization & Aho-Corasick Automaton (src/normalizer.ts, src/aho-corasick.ts)Copy Anchor Link

• TextNormalizer: Normalizes Unicode NFC/NFD, removes zero-width characters, unifies leetspeak, and translates Cyrillic/Greek homoglyphs to ASCII.
• AhoCorasickAutomaton: High-speed string matching automaton constructing root, node links, and failure pointers. Inspects text in a single O(M) pass where M is the text length.
• maskMatchedText(text, matches, options): Replaces sensitive or blocked words with masked tokens (***).

2. 64-Bit SimHash Near-Duplicate Deduplication (src/dedupe/)Copy Anchor Link

• computeSimHashBigInt(text, options) & computeSimHash(text): Produces a 64-bit fingerprint using tokenized 3-grams and 4-grams.
• hammingDistance(hashA, hashB): Computes bitwise difference (popcount(hashA ^ hashB)).
• isNearDuplicate(hashA, hashB, threshold): Determines whether two texts are semantic duplicates (typically <= 3 bit difference).
• NearDuplicateIndex: Divides 64-bit hashes into four 16-bit buckets using the Pigeonhole Principle for instant O(1) duplicate lookups among tens of thousands of stored documents.

3. Velocity & Heuristics Engine (src/velocity.ts, src/heuristics.ts)Copy Anchor Link

• SpamVelocityGuard: In-memory sliding-window limiter preventing bot floods from identical IPs or user IDs.
• scanHeuristics(text): Calculates textual entropy, uppercase percentage, repetitive punctuation, and suspicious outbound URL densities (Telegram, Zalo groups, URL shorteners).

4. Content Refinery (src/content-refinery/)Copy Anchor Link

• parseRssOrAtomXml(xml): Zero-dependency RSS and Atom XML feed parser.
• evaluateScrapedArticleQuality(input): Scores extracted articles for content completeness, headline spam, and advertising noise.

💻 Production Implementation & Code SnippetCopy Anchor Link

Anti-Spam & Near-Duplicate Verification ServiceCopy Anchor Link

import { 
  AntiSpamEngine, 
  NearDuplicateIndex, 
  computeSimHashBigInt,
  TextNormalizer,
  VIETNAMESE_STOPWORDS
} from '@kido/anti-spam';

// 1. Initialize Anti-Spam Engine with Rules & Presets
export const spamEngine = new AntiSpamEngine({
  checkHeuristicsByDefault: true,
  defaultRules: [
    { id: '1', keyword: 'casino online', action: 'BLOCK' },
    { id: '2', keyword: 'kéo tài khoản', action: 'BLOCK' },
    { id: '3', keyword: 'zalo.me/g/', action: 'BLOCK' },
    { id: '4', keyword: 't.me/', action: 'FLAG' },
  ],
});

// 2. Initialize In-Memory 64-Bit Near-Duplicate Index
const dedupeIndex = new NearDuplicateIndex({
  maxEntries: 50000,
  hammingThreshold: 3,
});

// 3. Comment Moderation Handler
export async function moderateComment(commentText: string, authorId: string) {
  // Step A: Fast-path Aho-Corasick & Heuristics Check (< 0.2ms)
  const validation = await spamEngine.validate(commentText, {
    authorId,
    strictScunthorpeGuard: true,
  });

  if (!validation.isValid) {
    return {
      allowed: false,
      reason: 'KEYWORD_OR_HEURISTIC_VIOLATION',
      blockedWords: validation.blockedWords,
    };
  }

  // Step B: 64-Bit SimHash Near-Duplicate Check
  const commentHash = computeSimHashBigInt(commentText, {
    stopwords: VIETNAMESE_STOPWORDS,
  });

  const dupMatch = dedupeIndex.findNearDuplicate(commentHash);
  if (dupMatch) {
    return {
      allowed: false,
      reason: 'DUPLICATE_SPAM_FLOOD',
      matchedWithId: dupMatch.id,
      hammingDistance: dupMatch.distance,
    };
  }

  // Step C: Record clean hash in dedupe index
  dedupeIndex.insert(`comment_${Date.now()}`, commentHash);

  return {
    allowed: true,
    maskedText: validation.maskedText ?? commentText,
  };
}

⚡ Benchmark & Verified k6 PerformanceCopy Anchor Link

Stress-tested on a single vCPU against 1,000 blacklisted phrases and 5,000-word articles:

🚀 Surgical Extraction & CLI InstallationCopy Anchor Link

# Install directly via Kido CLI
npx @kido/cli add @kido/anti-spam