@kido/sitemap-incrementalCopy Anchor Link

Architecture Paradigm: Cold/Hot Archive Partitioning + Atomic .tmp Safe Writes + Pre-Compressed Level-9 Gzip + Zero-Copy Kernel sendfile Serving.

💥 The Real-World Bottleneck (Why It Exists)Copy Anchor Link

In high-volume content, media, and e-commerce websites with 100,000 to 10,000,000 URLs:

1. The 504 Gateway Timeout Crawler Crash:
   • Web frameworks (Next.js App Router, Nuxt, Remix) frequently generate XML sitemaps dynamically on-demand inside SSR route handlers (app/sitemap.ts).
   • When Googlebot, Bingbot, and Yandex spider simultaneously, hundreds of concurrent sitemap requests execute heavy database table scans, serialize massive XML strings, and push CPU to 100%.
   • Result: Node.js event loops freeze, database connection pools are exhausted, and crawlers receive HTTP 504 Gateway Timeouts, causing immediate SEO ranking penalties.
2. The 95% Redundant Computation Waste:
   • In any mature website, over 95% of articles, historical blog posts, or past catalog items are immutable.
   • Re-querying and re-serializing 500 closed monthly sitemaps every night wastes millions of database read IOPS.
3. Corrupted Incomplete XML Reads:
   • Writing directly to destination XML files while crawlers are fetching them results in bots parsing truncated XML files, reporting false XML parse errors in Google Search Console.

🏛️ 3-Tier Production Evolution LadderCopy Anchor Link

📐 Architecture & Data FlowCopy Anchor Link

📦 Architecture & Core API ExportsCopy Anchor Link

All capabilities are strictly isolated in packages/sitemap-incremental:

1. src/index.tsCopy Anchor Link

• IncrementalSitemapEngine: Master orchestrator coordinating partitioning, database adapter fetching, XML building, and disk writing.
• buildUrlSetXml(urls, options): Formats standard <urlset> XML conforming to sitemaps.org schemas, with multi-locale xhtml:link alternates.
• buildSitemapIndexXml(sitemaps): Builds root <sitemapindex> listing all weekly and thematic child sitemaps.
• escapeXml(string): High-performance XML character escaping (&, <, >, ", ').
• PostgresSourceAdapter: Database adapter querying delta records based on updated_at timestamps.
• AtomicWriter: Disk writer handling .tmp file buffering and atomic rename operations.

💻 Production Implementation & Code SnippetCopy Anchor Link

1. Incremental Sitemap Generator Job (scripts/generate-sitemaps.ts)Copy Anchor Link

import {
  IncrementalSitemapEngine,
  buildUrlSetXml,
  buildSitemapIndexXml,
} from '@kido/sitemap-incremental';
import { Pool } from 'pg';

const dbPool = new Pool({
  connectionString: process.env.DATABASE_URL,
});

async function main() {
  console.log('[Sitemap] Starting incremental generation run...');

  // Initialize engine with Cold/Hot Partitioning enabled
  const engine = new IncrementalSitemapEngine({
    outDir: '/var/www/static/sitemaps',
    baseUrl: 'https://example.com',
    locales: ['vi', 'en'],
    isIncremental: true,  // Automatically skips closed historical weeks
    isGzipEnabled: true,   // Emits .xml.gz alongside .xml for zero-CPU delivery
    itemsPerSitemap: 10_000,
    fetchRecords: async ({ sinceDate, limit, offset }) => {
      // Delta query: only fetch records modified since the target window
      const query = `
        SELECT slug, updated_at, priority, changefreq
        FROM articles
        WHERE updated_at >= $1 AND published = true
        ORDER BY updated_at DESC
        LIMIT $2 OFFSET $3
      `;
      const result = await dbPool.query(query, [sinceDate, limit, offset]);
      return result.rows.map((row) => ({
        loc: `https://example.com/posts/${row.slug}`,
        lastmod: row.updated_at.toISOString(),
        changefreq: row.changefreq || 'weekly',
        priority: row.priority || 0.8,
      }));
    },
  });

  const summary = await engine.run();
  console.log(`[Sitemap] Finished in ${summary.elapsedMs}ms:`);
  console.log(`  - Fresh files generated: ${summary.totalFilesGenerated}`);
  console.log(`  - Cold files skipped: ${summary.totalFilesSkipped}`);
}

main().catch(console.error);

2. High-Performance Nginx Zero-Copy Configuration (nginx.conf)Copy Anchor Link

location /sitemaps/ {
    alias /var/www/static/sitemaps/;
    
    # 1. Zero-CPU pre-gzipped file delivery
    gzip_static on;
    
    # 2. Linux kernel zero-copy transfer directly from disk buffer
    sendfile on;
    tcp_nopush on;
    
    # 3. Aggressive caching headers for crawlers
    expires 1d;
    add_header Cache-Control "public, no-transform";
    
    # 4. Graceful fallback if a dynamic partition is pending
    try_files $uri $uri/ /api/sitemap-fallback;
}

⚡ Benchmark & Verified PerformanceCopy Anchor Link

Benchmarked with k6 simulating 2,000 concurrent crawler requests on a 100,000 URL database:

🚀 Surgical Extraction & CLI InstallationCopy Anchor Link

npx @kido/cli add sitemap-incremental