# Politrack – clean-room design > Written as if no code existed. Inputs were only the requirements: track what Austrian > politicians (federal → municipal) do, from many heterogeneous public sources, legally, with > every fact backed by links; user-selectable trust & topics; person/party overviews; calendar > and timeline; multilingual (de default, en); agent/RAG access; runs on an Unraid homelab in > Docker; scrapers deployable separately; easy for contributors to extend. > > [02-comparison-and-decisions.md](02-comparison-and-decisions.md) compares this design with > the first implementation and records which decisions won. ## 1. Quality goals (in priority order) 1. **Trustworthiness** – no fact without provenance; never present something as more certain than its sources; sensitive facts (criminal matters) are gated. Precision beats recall. 2. **Legal compliance by construction** – the framework, not the individual scraper author, enforces robots.txt, TDM reservations, rate limits, storage limits and data minimisation. 3. **Reproducibility** – any derived row can be explained ("which source, which response, which parser version") and regenerated from archived raw data after a parser fix, without hitting the source again. 4. **Extensibility** – adding a source must touch exactly one module (+ a fixture-based test and a docs line). No edits to SQL seeds, schedulers or registries elsewhere. 5. **Operability on a homelab** – few moving parts (one database, one image), self-healing schedules, everything inspectable with SQL. ## 2. System context ``` ┌───────────── public sources (APIs, open data, feeds, sitemaps, archives) ─────────────┐ ▼ │ ┌────────────────┐ Request ┌──────────┐ Response ┌──────────┐ Records ┌────────┐ │ │ Adapter.start()│──────────▶ │ Fetcher │─────────▶ │ Adapter. │───────▶ │ Loader │──▶ Postgres│ │ (per source) │◀────────── │ (policy, │ archive │ parse() │ FollowUp│ │ │ └────────────────┘ FollowUp │ archive) │──▶ Raw └──────────┘ └────────┘ │ └──────────┘ Archive (blobs + fetch_log) ▲ │ │ Records │ Enrichers (linking, events, legal, topics, ─────┘ │ corroboration, embeddings) read Postgres │ │ Serving: FastAPI (typed JSON, OpenAPI) ─▶ Next.js web (de/en) MCP server ─▶ AI agents │ read-only SQL role ─▶ notebooks/BI/agents pgvector chunks ─▶ RAG │ ``` Four strictly separated stages: | Stage | Input → output | Side effects | Determinism | |---|---|---|---| | **Fetch** | `Request` → `Response` | network; raw archive write | no | | **Parse** | `Response` → `Record`s and follow-up `Request`s | none (pure) | yes, versioned | | **Load** | `Record` → canonical rows | DB writes (single code path) | yes, idempotent | | **Enrich** | canonical rows → `Record`s | via Loader only | yes, idempotent | Because parse is pure and responses are archived, `politrack reparse ` rebuilds data after a parser fix without re-fetching. Because load is the only writer, invariants (no claim without evidence, data minimisation, identity rules) live in one place. ## 3. Data model ### 3.1 Principles * **Claim/evidence**: every user-visible fact is a `claim` with ≥1 `claim_evidence` row pointing to a `document` (a URL) published by a `source`. Claims from different sources about the same fact collapse into one claim with several evidence rows ("corroborated"). * **Typed facets**: claims of some types carry a structured facet row (role, vote position, money flow, event participation, statement, legal proceeding). Facets make analytics and correlations possible; the claim makes provenance and display uniform. * **Identifiers instead of ID columns**: entities get any number of external identifiers in one table (`scheme`, `value`), e.g. `parlament:2339`, `wikidata:Q123`, `ep:197479`, `bluesky:did:plc:…`, `telegram:HKickl`, `uid:ATU12345678`, `rtr:7846`. New sources add schemes, not columns. * **Bitemporal enough**: claims have `occurred_at` (point events) or `valid_from/valid_to` (states such as mandates); documents have `published_at`; everything has `created_at/updated_at`. ### 3.2 Tables (abridged; full DDL in `backend/migrations`) ``` source(slug PK-ish, name, kind, homepage, license, access, trust, store_fulltext, keep_raw, tos_notes, …) fetch(id, adapter, source_id, request_fp, url, method, status, sha256, size, content_type, etag, last_modified, fetched_at, request jsonb, parsed_sha256, parser_version) -- raw archive index document(id, source_id, url UNIQUE, title, doc_type, lang, published_at, excerpt, full_text, fetch_id, story_id, meta jsonb) person(id, slug, full_name, given_name, family_name, prefixes…, birth_date, death_date, gender, photo_url) organization(id, slug, name, short_name, kind, parent_id, color, …) identifier(entity_type, entity_id, scheme, value, source_slug, verified) UNIQUE(entity_type, scheme, value) alias(entity_type, entity_id, alias, alias_norm) social_account(id, person_id|organization_id, platform, handle, url, verified, source_slug) claim(id, claim_type, fingerprint UNIQUE, person_id, organization_id, counterpart_person_id, counterpart_org_id, occurred_at, valid_from, valid_to, summary_de, summary_en, data jsonb, status, sensitive, review_status, merged_into, created_by, …) claim_evidence(claim_id, document_id, stance, quote, locator, extracted_by, confidence) role / vote_event / vote_position / money_flow / event / event_participant / statement / legal_proceeding -- facets topic, topic_assignment, story, chunk(text, tsv, embedding vector(1024)) run(id, adapter, started_at, finished_at, status, stats, error), state(adapter, key, value) ``` ### 3.3 Claim identity The *Loader* (not the scraper) computes a claim's fingerprint from **resolved entity ids** and the semantically identifying fields of its type: | claim type | identity | |---|---| | role | person, body, canonical role, start date | | vote | vote event key, voter (person or party) | | donation / money_flow | kind, payer, payee, amount, date, external reference | | statement | person, URL | | media_mention | URL, subject | | meeting / travel | person, anchor (counterpart surname or country), ISO week | | legal_proceeding, others | explicit key from the adapter | Using resolved ids means a Wikidata mandate and a Parliament mandate of the same person, body and start date *are the same claim* by construction. A fuzzy **corroboration** pass handles near-duplicates (dates off by days, wording differences) by merging claims (`merged_into`). ### 3.4 Entity resolution Records refer to entities through `PersonRef` / `OrgRef`: ```python PersonRef(ids={"parlament": "2339"}, name="Selma Yildirim") # strong id PersonRef(name="Auer", hints={"surname_in_body": "nationalrat", "on": date(2025, 9, 24)}) OrgRef(name="Hypo Tirol Bank AG", kind="company", ids={"uid": "ATU…"}) ``` Resolution order: (1) any strong identifier → entity; (2) strategy named in hints (e.g. *surname among members of body on date*); (3) exact normalised alias; (4) for organisations: normalised legal name (legal-form suffixes stripped); (5) create — persons are only created by "authoritative" adapters (`creates_persons=True`), other adapters leave unresolvable person refs unlinked and count them. Ambiguity never guesses. ## 4. The adapter framework (the nitty-gritty) ### 4.1 SourceSpec – single source of truth for a publisher ```python @dataclass(frozen=True) class SourceSpec: slug: str # "parlament" name: str kind: Literal["official", "ngo", "media", "social", "aggregator", "user"] homepage: str license: str access: Literal["api", "bulk", "rss", "scrape", "manual"] trust: int # default 0–100, editorial store_fulltext: bool # may we keep article/body text? keep_raw: bool = True # may we archive raw responses? (False for TDM-reserved) min_interval: float = 2.0 # seconds between requests to one host robots_exempt: tuple[str, ...] = () # URL prefixes of documented APIs; justify in tos_notes tos_notes: str = "" ``` Specs live next to their adapter. `politrack sources sync` (run by `migrate`) upserts them into the `source` table, so SQL never needs editing to add a source. ### 4.2 Adapter ```python class Adapter: spec: ClassVar[SourceSpec] name: ClassVar[str] # "parlament.items" (a source may have several adapters) schedule: ClassVar[str | None] # cron (UTC) or None = manual only queue: ClassVar[str] = "scrape" parser_version: ClassVar[int] = 1 # bump when parse() output changes → reparse def start(self, ctx: RunContext) -> Iterable[Request]: ... def parse(self, ctx: RunContext, response: Response) -> Iterable[Record | Request]: ... ``` * `start()` yields the entry requests (list endpoints, index pages, feeds). It may read `ctx.state` (cursors) and `ctx.since`. * `parse()` is **pure**: no DB access, no network. It yields records and follow-up requests (detail pages). Dispatch on `response.request.kind` (`"list"`, `"detail"`, …). * Anything that needs the database (e.g. "who sat in the Nationalrat on that day") is expressed as resolution hints on refs and done by the Loader. ### 4.3 Request / Response ```python @dataclass(frozen=True) class Request: url: str kind: str = "default" # parse() dispatch key method: str = "GET" params: dict | None = None json: Any = None headers: dict | None = None meta: dict = field(default_factory=dict) # passed through to parse (e.g. {"gp": "XXVIII"}) conditional: bool = True # send If-None-Match / If-Modified-Since skip_unchanged: bool = True # don't reload records if body hash and parser_version unchanged @dataclass class Response: request: Request status: int url: str headers: Mapping[str, str] body: bytes fetched_at: datetime sha256: str fetch_id: int | None # row in the raw archive index def text(self) -> str; def json(self) -> Any; def soup(self) -> BeautifulSoup ``` ### 4.4 Fetcher (policy enforcement) For every request the fetcher: 1. checks robots.txt (cached per origin) unless the URL starts with a `robots_exempt` prefix; 2. throttles per host (`max(spec.min_interval, robots crawl-delay)`); 3. sends an honest User-Agent with contact address, and conditional headers from the last fetch; 4. retries 429/5xx with exponential back-off, honouring `Retry-After`; 5. detects TDM reservations (`tdm-reservation` header, `/.well-known/tdmrep.json`) and records them on the response (`response.tdm_reserved`) – the Loader then drops full text; 6. writes the body to the raw archive (`sha256`-addressed, gzip) **iff** `spec.keep_raw` and no TDM reservation, and logs a `fetch_log` row in every case. ### 4.5 Records (typed, validated) ```python Evidence(url, title=None, doc_type="web_page", published_at=None, excerpt=None, full_text=None, quote=None, locator="", stance="supports", confidence=1.0, source=None) PersonRecord(ref, full_name, given_name=None, family_name=None, …, aliases=(), social=()) OrgRecord(ref, name, short_name=None, kind="other", parent=None, color=None) Claim(claim_type, summary_de, evidence, person=None, org=None, counterpart_person=None, counterpart_org=None, occurred_at=None, valid_from=None, valid_to=None, summary_en=None, data={}, topics=(), sensitive=False, review=False, key=None, facet=None) # facets Role(role, level, body=None, party=None, region=None, start=None, end=None) VoteEvent(key, chamber, title, voted_on, result, stage=None, item_ref=None, roll_call=False, url=None, …) VotePosition(event_key, position) # voter = claim.person or claim.org Money(kind, amount, currency="EUR", payer=None, payee=None, payer_name=None, payee_name=None, date=None, reference=None, category=None, period=None) Event(key, kind, title, starts_at, ends_at=None, location=None, country=None, url=None, body=None) Participation(event_key, role="participant") Statement(context, url, said_at=None, medium=None, title=None, excerpt=None, platform=None) Legal(stage, offense=None, authority=None, case_ref=None, stage_date=None) Document(evidence) # a document without a claim (e.g. a headline); # enrichers create claims from it later ``` Data minimisation is enforced by *schema*: e.g. `Money` has no address field, so a parser cannot leak a donor's street address into the database even by accident. ### 4.6 Runner ```python def run(adapter, mode="fetch", since=None): ctx = RunContext(adapter, since, state=StateStore(adapter.name), stats=Counter()) queue = deque(adapter.start(ctx)); seen = set() while queue: req = queue.popleft() if fingerprint(req) in seen: continue; seen.add(fingerprint(req)) resp = fetcher.fetch(req) if mode == "fetch" else archive.latest(req) if resp is None: continue if req.skip_unchanged and archive.already_parsed(resp, adapter.parser_version): # records already loaded; still follow links so an interrupted run's children get fetched queue.extend(r for r in adapter.parse(ctx, resp) if isinstance(r, Request)); continue with db.transaction(): # one transaction per response for item in adapter.parse(ctx, resp): queue.append(item) if isinstance(item, Request) else loader.load(item, ctx) archive.mark_parsed(resp, adapter.parser_version) record run row with stats/errors ``` Per-response transactions mean a crash loses one page, never the whole run; errors in one response are recorded and the run continues (error budget → run status `partial`). ### 4.7 Scheduling & deployment * Procrastinate (Postgres-backed queue): the worker builds one periodic task per adapter from `adapter.schedule`/`adapter.queue` – no hand-written task list. * Queues `scrape`, `social`, `bulk`, `enrich` let heavy bulk jobs (procurement, ad archives) run on their own worker (even another machine) without blocking light ones. * One Docker image for api/worker/mcp/migrate; third-party adapter packages register via the Python entry-point group `politrack.adapters`, so a separate deployment can add sources without forking. ### 4.8 Enrichers Same Loader, different input: an `Enricher` has `run(ctx) -> Iterable[Record]` over database rows (documents not yet linked, claims not yet tagged…). Built-ins: mention linker, event extractor (trips/meetings), legal-matter detector (→ review queue), topic tagger, corroborator (merge near-duplicates, cluster news stories), say-vs-vote matcher, chunker + embedder. ## 5. Serving * **API**: FastAPI with Pydantic response models (OpenAPI is the contract). Standard filters everywhere: `person`, `party`, `types`, `topics`, `since`, `until`, `exclude_sources`, `min_trust`, `limit/offset`. * **Web**: Next.js App Router + next-intl (de at `/`, en at `/en`); TypeScript types generated from OpenAPI (`politrack openapi --ts`); preferences in a cookie so server components can filter; a **renderer registry** maps claim types to components (adding a claim type = one component). * **Agents**: MCP tools mirroring the API; read-only SQL role; `COMMENT ON` everything; `claim_with_evidence` view; `chunk` table for RAG with citations. ## 6. Testing strategy | Level | What | How | |---|---|---| | Parser golden tests | each adapter's `parse()` | real responses saved as fixtures (`tests/fixtures//…`), expected records as JSON next to them; `UPDATE_GOLDEN=1` regenerates | | Unit tests | text normalisation, resolvers, identity | plain pytest | | Loader/integration | load records into an ephemeral Postgres, assert rows | `pytest -m integration` with `POLITRACK_TEST_DATABASE_URL` | | API smoke | every endpoint against a seeded DB | FastAPI TestClient | | Frontend | type check + build | `npm run typecheck && npm run build` | ## 7. Legal & ethical guard-rails in code * Fetcher enforces robots/TDM/rate limits; `keep_raw`/`store_fulltext` per source. * Record schemas exclude data we must not keep (addresses, private contact data). * `sensitive=True` claims render the presumption-of-innocence notice unless stage is an undisputed final conviction; auto-detected legal matters require `review`. * Messengers: only *public broadcast channels* (e.g. public Telegram channels' web preview); never private groups, never platforms whose terms forbid automated access (WhatsApp, X without API, Facebook/Instagram). ## 8. Operations * Services: `db` (Postgres 16 + pgvector), `migrate` (one-shot), `api`, `web`, `mcp`, `worker-scrape`, `worker-bulk`, `worker-enrich`, optional `ollama`. * Volumes: Postgres data, raw archive (`/data/raw`), Ollama models. * Observability: `run` table, `politrack status`, `/api/v1/admin/runs` (token), structured logs. * Backups: `pg_dump` + raw archive rsync (raw archive is optional to back up – it is a cache that can be re-fetched, but it is also the audit trail).