politics-tracking/docs/architecture/01-design-from-scratch.md

Politrack – clean-room design

Written as if no code existed. Inputs were only the requirements: track what Austrian politicians (federal → municipal) do, from many heterogeneous public sources, legally, with every fact backed by links; user-selectable trust & topics; person/party overviews; calendar and timeline; multilingual (de default, en); agent/RAG access; runs on an Unraid homelab in Docker; scrapers deployable separately; easy for contributors to extend.

02-comparison-and-decisions.md compares this design with the first implementation and records which decisions won.

1. Quality goals (in priority order)

1. Trustworthiness – no fact without provenance; never present something as more certain than its sources; sensitive facts (criminal matters) are gated. Precision beats recall.
2. Legal compliance by construction – the framework, not the individual scraper author, enforces robots.txt, TDM reservations, rate limits, storage limits and data minimisation.
3. Reproducibility – any derived row can be explained ("which source, which response, which parser version") and regenerated from archived raw data after a parser fix, without hitting the source again.
4. Extensibility – adding a source must touch exactly one module (+ a fixture-based test and a docs line). No edits to SQL seeds, schedulers or registries elsewhere.
5. Operability on a homelab – few moving parts (one database, one image), self-healing schedules, everything inspectable with SQL.

2. System context

               ┌───────────── public sources (APIs, open data, feeds, sitemaps, archives) ─────────────┐
               ▼                                                                                        │
      ┌────────────────┐  Request   ┌──────────┐ Response  ┌──────────┐ Records ┌────────┐            │
      │ Adapter.start()│──────────▶ │ Fetcher  │─────────▶ │ Adapter. │───────▶ │ Loader │──▶ Postgres│
      │ (per source)   │◀────────── │ (policy, │  archive  │ parse()  │ FollowUp│        │            │
      └────────────────┘  FollowUp  │ archive) │──▶ Raw    └──────────┘         └────────┘            │
                                    └──────────┘   Archive (blobs + fetch_log)      ▲                 │
                                                                                      │ Records         │
                                      Enrichers (linking, events, legal, topics, ─────┘                 │
                                      corroboration, embeddings) read Postgres                          │
                                                                                                        │
      Serving: FastAPI (typed JSON, OpenAPI) ─▶ Next.js web (de/en)      MCP server ─▶ AI agents       │
               read-only SQL role ─▶ notebooks/BI/agents                 pgvector chunks ─▶ RAG         │

Four strictly separated stages:

Stage	Input → output	Side effects	Determinism
Fetch	Request → Response	network; raw archive write	no
Parse	Response → Records and follow-up Requests	none (pure)	yes, versioned
Load	Record → canonical rows	DB writes (single code path)	yes, idempotent
Enrich	canonical rows → Records	via Loader only	yes, idempotent

Because parse is pure and responses are archived, politrack reparse <adapter> rebuilds data after a parser fix without re-fetching. Because load is the only writer, invariants (no claim without evidence, data minimisation, identity rules) live in one place.

3. Data model

3.1 Principles

• Claim/evidence: every user-visible fact is a claim with ≥1 claim_evidence row pointing to a document (a URL) published by a source. Claims from different sources about the same fact collapse into one claim with several evidence rows ("corroborated").
• Typed facets: claims of some types carry a structured facet row (role, vote position, money flow, event participation, statement, legal proceeding). Facets make analytics and correlations possible; the claim makes provenance and display uniform.
• Identifiers instead of ID columns: entities get any number of external identifiers in one table (scheme, value), e.g. parlament:2339, wikidata:Q123, ep:197479, bluesky:did:plc:…, telegram:HKickl, uid:ATU12345678, rtr:7846. New sources add schemes, not columns.
• Bitemporal enough: claims have occurred_at (point events) or valid_from/valid_to (states such as mandates); documents have published_at; everything has created_at/updated_at.

3.2 Tables (abridged; full DDL in backend/migrations)

source(slug PK-ish, name, kind, homepage, license, access, trust, store_fulltext, keep_raw, tos_notes, …)
fetch(id, adapter, source_id, request_fp, url, method, status, sha256, size, content_type,
      etag, last_modified, fetched_at, request jsonb, parsed_sha256, parser_version)      -- raw archive index
document(id, source_id, url UNIQUE, title, doc_type, lang, published_at, excerpt, full_text,
         fetch_id, story_id, meta jsonb)
person(id, slug, full_name, given_name, family_name, prefixes…, birth_date, death_date, gender, photo_url)
organization(id, slug, name, short_name, kind, parent_id, color, …)
identifier(entity_type, entity_id, scheme, value, source_slug, verified)  UNIQUE(entity_type, scheme, value)
alias(entity_type, entity_id, alias, alias_norm)
social_account(id, person_id|organization_id, platform, handle, url, verified, source_slug)
claim(id, claim_type, fingerprint UNIQUE, person_id, organization_id, counterpart_person_id,
      counterpart_org_id, occurred_at, valid_from, valid_to, summary_de, summary_en, data jsonb,
      status, sensitive, review_status, merged_into, created_by, …)
claim_evidence(claim_id, document_id, stance, quote, locator, extracted_by, confidence)
role / vote_event / vote_position / money_flow / event / event_participant / statement /
legal_proceeding                                                             -- facets
topic, topic_assignment, story, chunk(text, tsv, embedding vector(1024))
run(id, adapter, started_at, finished_at, status, stats, error), state(adapter, key, value)

3.3 Claim identity

The Loader (not the scraper) computes a claim's fingerprint from resolved entity ids and the semantically identifying fields of its type:

claim type	identity
role	person, body, canonical role, start date
vote	vote event key, voter (person or party)
donation / money_flow	kind, payer, payee, amount, date, external reference
statement	person, URL
media_mention	URL, subject
meeting / travel	person, anchor (counterpart surname or country), ISO week
legal_proceeding, others	explicit key from the adapter

Using resolved ids means a Wikidata mandate and a Parliament mandate of the same person, body and start date are the same claim by construction. A fuzzy corroboration pass handles near-duplicates (dates off by days, wording differences) by merging claims (merged_into).

3.4 Entity resolution

Records refer to entities through PersonRef / OrgRef:

PersonRef(ids={"parlament": "2339"}, name="Selma Yildirim")             # strong id
PersonRef(name="Auer", hints={"surname_in_body": "nationalrat", "on": date(2025, 9, 24)})
OrgRef(name="Hypo Tirol Bank AG", kind="company", ids={"uid": "ATU…"})

Resolution order: (1) any strong identifier → entity; (2) strategy named in hints (e.g. surname among members of body on date); (3) exact normalised alias; (4) for organisations: normalised legal name (legal-form suffixes stripped); (5) create — persons are only created by "authoritative" adapters (creates_persons=True), other adapters leave unresolvable person refs unlinked and count them. Ambiguity never guesses.

4. The adapter framework (the nitty-gritty)

4.1 SourceSpec – single source of truth for a publisher

@dataclass(frozen=True)
class SourceSpec:
    slug: str                 # "parlament"
    name: str
    kind: Literal["official", "ngo", "media", "social", "aggregator", "user"]
    homepage: str
    license: str
    access: Literal["api", "bulk", "rss", "scrape", "manual"]
    trust: int                # default 0–100, editorial
    store_fulltext: bool      # may we keep article/body text?
    keep_raw: bool = True     # may we archive raw responses? (False for TDM-reserved)
    min_interval: float = 2.0 # seconds between requests to one host
    robots_exempt: tuple[str, ...] = ()   # URL prefixes of documented APIs; justify in tos_notes
    tos_notes: str = ""

Specs live next to their adapter. politrack sources sync (run by migrate) upserts them into the source table, so SQL never needs editing to add a source.

4.2 Adapter

class Adapter:
    spec: ClassVar[SourceSpec]
    name: ClassVar[str]                 # "parlament.items" (a source may have several adapters)
    schedule: ClassVar[str | None]      # cron (UTC) or None = manual only
    queue: ClassVar[str] = "scrape"
    parser_version: ClassVar[int] = 1   # bump when parse() output changes → reparse

    def start(self, ctx: RunContext) -> Iterable[Request]: ...
    def parse(self, ctx: RunContext, response: Response) -> Iterable[Record | Request]: ...

• start() yields the entry requests (list endpoints, index pages, feeds). It may read ctx.state (cursors) and ctx.since.
• parse() is pure: no DB access, no network. It yields records and follow-up requests (detail pages). Dispatch on response.request.kind ("list", "detail", …).
• Anything that needs the database (e.g. "who sat in the Nationalrat on that day") is expressed as resolution hints on refs and done by the Loader.

4.3 Request / Response

@dataclass(frozen=True)
class Request:
    url: str
    kind: str = "default"           # parse() dispatch key
    method: str = "GET"
    params: dict | None = None
    json: Any = None
    headers: dict | None = None
    meta: dict = field(default_factory=dict)   # passed through to parse (e.g. {"gp": "XXVIII"})
    conditional: bool = True        # send If-None-Match / If-Modified-Since
    skip_unchanged: bool = True     # don't reload records if body hash and parser_version unchanged

@dataclass
class Response:
    request: Request
    status: int
    url: str
    headers: Mapping[str, str]
    body: bytes
    fetched_at: datetime
    sha256: str
    fetch_id: int | None            # row in the raw archive index
    def text(self) -> str; def json(self) -> Any; def soup(self) -> BeautifulSoup

4.4 Fetcher (policy enforcement)

For every request the fetcher:

1. checks robots.txt (cached per origin) unless the URL starts with a robots_exempt prefix;
2. throttles per host (max(spec.min_interval, robots crawl-delay));
3. sends an honest User-Agent with contact address, and conditional headers from the last fetch;
4. retries 429/5xx with exponential back-off, honouring Retry-After;
5. detects TDM reservations (tdm-reservation header, /.well-known/tdmrep.json) and records them on the response (response.tdm_reserved) – the Loader then drops full text;
6. writes the body to the raw archive (sha256-addressed, gzip) iff spec.keep_raw and no TDM reservation, and logs a fetch_log row in every case.

4.5 Records (typed, validated)

Evidence(url, title=None, doc_type="web_page", published_at=None, excerpt=None, full_text=None,
         quote=None, locator="", stance="supports", confidence=1.0, source=None)
PersonRecord(ref, full_name, given_name=None, family_name=None, …, aliases=(), social=())
OrgRecord(ref, name, short_name=None, kind="other", parent=None, color=None)
Claim(claim_type, summary_de, evidence, person=None, org=None, counterpart_person=None,
      counterpart_org=None, occurred_at=None, valid_from=None, valid_to=None, summary_en=None,
      data={}, topics=(), sensitive=False, review=False, key=None, facet=None)
# facets
Role(role, level, body=None, party=None, region=None, start=None, end=None)
VoteEvent(key, chamber, title, voted_on, result, stage=None, item_ref=None, roll_call=False, url=None, …)
VotePosition(event_key, position)            # voter = claim.person or claim.org
Money(kind, amount, currency="EUR", payer=None, payee=None, payer_name=None, payee_name=None,
      date=None, reference=None, category=None, period=None)
Event(key, kind, title, starts_at, ends_at=None, location=None, country=None, url=None, body=None)
Participation(event_key, role="participant")
Statement(context, url, said_at=None, medium=None, title=None, excerpt=None, platform=None)
Legal(stage, offense=None, authority=None, case_ref=None, stage_date=None)
Document(evidence)                           # a document without a claim (e.g. a headline);
                                             # enrichers create claims from it later

Data minimisation is enforced by schema: e.g. Money has no address field, so a parser cannot leak a donor's street address into the database even by accident.

4.6 Runner

def run(adapter, mode="fetch", since=None):
    ctx = RunContext(adapter, since, state=StateStore(adapter.name), stats=Counter())
    queue = deque(adapter.start(ctx)); seen = set()
    while queue:
        req = queue.popleft()
        if fingerprint(req) in seen: continue; seen.add(fingerprint(req))
        resp = fetcher.fetch(req) if mode == "fetch" else archive.latest(req)
        if resp is None: continue
        if req.skip_unchanged and archive.already_parsed(resp, adapter.parser_version):
            # records already loaded; still follow links so an interrupted run's children get fetched
            queue.extend(r for r in adapter.parse(ctx, resp) if isinstance(r, Request)); continue
        with db.transaction():                 # one transaction per response
            for item in adapter.parse(ctx, resp):
                queue.append(item) if isinstance(item, Request) else loader.load(item, ctx)
            archive.mark_parsed(resp, adapter.parser_version)
    record run row with stats/errors

Per-response transactions mean a crash loses one page, never the whole run; errors in one response are recorded and the run continues (error budget → run status partial).

4.7 Scheduling & deployment

• Procrastinate (Postgres-backed queue): the worker builds one periodic task per adapter from adapter.schedule/adapter.queue – no hand-written task list.
• Queues scrape, social, bulk, enrich let heavy bulk jobs (procurement, ad archives) run on their own worker (even another machine) without blocking light ones.
• One Docker image for api/worker/mcp/migrate; third-party adapter packages register via the Python entry-point group politrack.adapters, so a separate deployment can add sources without forking.

4.8 Enrichers

Same Loader, different input: an Enricher has run(ctx) -> Iterable[Record] over database rows (documents not yet linked, claims not yet tagged…). Built-ins: mention linker, event extractor (trips/meetings), legal-matter detector (→ review queue), topic tagger, corroborator (merge near-duplicates, cluster news stories), say-vs-vote matcher, chunker + embedder.

5. Serving

• API: FastAPI with Pydantic response models (OpenAPI is the contract). Standard filters everywhere: person, party, types, topics, since, until, exclude_sources, min_trust, limit/offset.
• Web: Next.js App Router + next-intl (de at /, en at /en); TypeScript types generated from OpenAPI (politrack openapi --ts); preferences in a cookie so server components can filter; a renderer registry maps claim types to components (adding a claim type = one component).
• Agents: MCP tools mirroring the API; read-only SQL role; COMMENT ON everything; claim_with_evidence view; chunk table for RAG with citations.

6. Testing strategy

Level	What	How
Parser golden tests	each adapter's parse()	real responses saved as fixtures (tests/fixtures/<adapter>/…), expected records as JSON next to them; UPDATE_GOLDEN=1 regenerates
Unit tests	text normalisation, resolvers, identity	plain pytest
Loader/integration	load records into an ephemeral Postgres, assert rows	pytest -m integration with POLITRACK_TEST_DATABASE_URL
API smoke	every endpoint against a seeded DB	FastAPI TestClient
Frontend	type check + build	npm run typecheck && npm run build

7. Legal & ethical guard-rails in code

• Fetcher enforces robots/TDM/rate limits; keep_raw/store_fulltext per source.
• Record schemas exclude data we must not keep (addresses, private contact data).
• sensitive=True claims render the presumption-of-innocence notice unless stage is an undisputed final conviction; auto-detected legal matters require review.
• Messengers: only public broadcast channels (e.g. public Telegram channels' web preview); never private groups, never platforms whose terms forbid automated access (WhatsApp, X without API, Facebook/Instagram).

8. Operations

• Services: db (Postgres 16 + pgvector), migrate (one-shot), api, web, mcp, worker-scrape, worker-bulk, worker-enrich, optional ollama.
• Volumes: Postgres data, raw archive (/data/raw), Ollama models.
• Observability: run table, politrack status, /api/v1/admin/runs (token), structured logs.
• Backups: pg_dump + raw archive rsync (raw archive is optional to back up – it is a cache that can be re-fetched, but it is also the audit trail).