← All loops · data · any

Scrape-and-validate loop

Every source is scraped, validated against schema, and loaded.

F1 · Goal

Every source is scraped, validated against schema, and loaded.

F2 · Trigger

*/30 * * * *

F3 · Discover

Sources list with last-success timestamps; pick stale ones.

F4 · Act

Fetch, parse, normalize; retry failures with exponential backoff.

F5 · Verify

Schema validation pass rate at least 98%; row counts within expected band.

F6 · Persist

Source state table: last_run, last_success, consecutive_failures.

F7 · Exit

All sources fresh, or a source fails 3x (quarantine and alert).

Cost

~6k tokens per cycle (measured)

Runnable code

*/30 * * * * cd ~/pipeline && claude -p "$(cat loops/scrape-validate.md)" --allowedTools "Bash,Write"

Adapted it for your stack? Submit your variant →