Content Library

A persistent inventory of a brand's own site pages — blog posts, guides, resources — crawled from the sitemap and AI-enriched with topics and a relevance score. Its whole job: give the planner and radar REAL URLs to link in posts.

How it works

Crawl sitemap

Fetch /sitemap.xml, handle nested sitemaps (1 level deep), filter to linkable content pages (drop homepage, locale roots, non-content routes like cart/login/privacy).

Fetch pages

Polite crawl at ~2 req/s (500ms delay). Extract title, meta description, body text (first 8000 chars). Skip pages with <200 chars of body.

AI-enrich

One batch Gemini call: for each page, extract 3-5 topic tags and a relevance score (0-100) — "how useful is this page to link from the brand's social posts?"

Upsert

Store in brand_pages with url_hash (SHA-1) as dedup key. Incremental: unchanged pages just get re-scored.

Relevance scoring

Product, blog, and guide pages that answer audience questions score high (70-100). Legal, generic, or thin pages score low (0-30). The planner only picks high-relevance pages.

The library avoids guessing each site's content taxonomy — it indexes everything from the sitemap and lets the AI relevance score decide what's actually linkable.

Usage tracking

When a post links one of the brand's pages, last_used_at is stamped. The query helper returns pages ordered by relevance DESC, then last_used_at ASC — so the same page isn't re-shared in a loop.

Constants: max 80 pages per scan, 500ms fetch delay, 8000 chars body, 12 pages surfaced to the planner.

When the brand publishes new content on their site, a re-scan picks it up. The planner automatically rotates through pages.

Endpoint: POST /api/v1/brands/:slug/library/scan — triggers a manual crawl (CLI auth required).