# Fetch Rules

Readable rules that govern how `daily_news_watcher.py` fetches and processes
sources. Keep this file legible to people and agents.

## Source Acceptance

- Only public URLs. Refuse `file://`, internal hostnames, and URLs that require
  authentication headers.
- Prefer `rss` and `atom` feeds. Fall back to `webpage` only when no feed is
  discoverable.
- Reject paywalled sources. If a publication requires login, do not attempt to
  bypass it; record the source as `webpage` and skip article extraction.

## Fetch Behavior

- Use a single identifying `User-Agent`:
  `daily-news-watcher/0.1 (+https://github.com/Prompthon-IO/agentic-lab)`.
- Timeout each request at 15 seconds. On timeout, record the failure on the
  `runs` row and continue with the next source.
- Cap each source at 25 articles per run to keep reports readable.
- Honor obvious rate limits: do not refetch a source whose `last_checked_at`
  is within the same minute.

## Filtering

- `--hours N` filters articles whose `published_at` falls inside the last N
  hours. If `published_at` is missing, fall back to `fetched_at`.
- `--topic T` is an inclusive case-insensitive match against the article
  title, summary, and the source's `tags` column.

## Deduplication

- URL canonical form: lowercase host, strip default ports, drop trailing
  slashes, remove `utm_*`, `gclid`, `fbclid`, `mc_cid`, `mc_eid` query params.
- Content hash: SHA-256 of the normalized `title + "\n" + summary`.
- Skip an article if either its canonical URL or its content hash already
  exists in the `articles` table.

## Summarization

- Extract a short snippet: strip HTML, collapse whitespace, take the first
  ~280 characters at a sentence boundary when possible.
- The deterministic snippet is intentionally rough. Codex is expected to
  rewrite the snippets in its final reply.

## Safety And Privacy

- Never store cookies, auth headers, or browser session data.
- Never bypass paywalls, captchas, or access controls.
- Local artifacts under `~/.codex/state/daily-news-watcher/` are not committed
  to git unless the user explicitly asks for sample artifacts.
- Failures are recorded per-source. One failing source must not abort the
  whole run.