Politrack: track Austrian politicians from public sources

Initial import of Politrack, a self-hosted service that tracks Austrian
politicians from federal to municipal level - roles, votes, money flows,
statements, meetings, social media and legal proceedings - with every claim
linked to its sources.

- backend: FastAPI + Postgres/pgvector; adapter framework (fetch -> parse ->
  load -> enrich) with robots.txt/TDM-aware fetching, a raw-response archive
  for reparsing, and a claim/evidence model with corroboration
- sources: Parliament, Wikidata, EU Parliament/Commission, Council of
  Ministers, Rechnungshof party reports and donations, procurement, RTR media
  transparency, political ad libraries, WKStA, news feeds, Wikipedia edits,
  Bluesky, Mastodon, public Telegram channels, YouTube and X (official API)
- web: Next.js frontend in German and English with typed API client
- mcp: MCP server for AI agents and RAG
- docs: architecture, data model, sources, legal notes, Unraid deployment and
  contributor guides; Docker Compose setup for Unraid

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
simonandClaude Opus 5.5 committed 2026-10-28 20:02:01 +01:00
commit 583ccba31c
187 files changed
+39890

No files matched your search

+325
View File
@@ -0,0 +1,325 @@
# Politrack – clean-room design
> Written as if no code existed. Inputs were only the requirements: track what Austrian
> politicians (federal → municipal) do, from many heterogeneous public sources, legally, with
> every fact backed by links; user-selectable trust & topics; person/party overviews; calendar
> and timeline; multilingual (de default, en); agent/RAG access; runs on an Unraid homelab in
> Docker; scrapers deployable separately; easy for contributors to extend.
>
> [02-comparison-and-decisions.md](02-comparison-and-decisions.md) compares this design with
> the first implementation and records which decisions won.
## 1. Quality goals (in priority order)
1. **Trustworthiness** – no fact without provenance; never present something as more certain
than its sources; sensitive facts (criminal matters) are gated. Precision beats recall.
2. **Legal compliance by construction** – the framework, not the individual scraper author,
enforces robots.txt, TDM reservations, rate limits, storage limits and data minimisation.
3. **Reproducibility** – any derived row can be explained ("which source, which response,
which parser version") and regenerated from archived raw data after a parser fix, without
hitting the source again.
4. **Extensibility** – adding a source must touch exactly one module (+ a fixture-based test
and a docs line). No edits to SQL seeds, schedulers or registries elsewhere.
5. **Operability on a homelab** – few moving parts (one database, one image), self-healing
schedules, everything inspectable with SQL.
## 2. System context
```
┌───────────── public sources (APIs, open data, feeds, sitemaps, archives) ─────────────┐
▼ │
┌────────────────┐ Request ┌──────────┐ Response ┌──────────┐ Records ┌────────┐ │
│ Adapter.start()│──────────▶ │ Fetcher │─────────▶ │ Adapter. │───────▶ │ Loader │──▶ Postgres│
│ (per source) │◀────────── │ (policy, │ archive │ parse() │ FollowUp│ │ │
└────────────────┘ FollowUp │ archive) │──▶ Raw └──────────┘ └────────┘ │
└──────────┘ Archive (blobs + fetch_log) ▲ │
│ Records │
Enrichers (linking, events, legal, topics, ─────┘ │
corroboration, embeddings) read Postgres │
│
Serving: FastAPI (typed JSON, OpenAPI) ─▶ Next.js web (de/en) MCP server ─▶ AI agents │
read-only SQL role ─▶ notebooks/BI/agents pgvector chunks ─▶ RAG │
```
Four strictly separated stages:
| Stage | Input → output | Side effects | Determinism |
|---|---|---|---|
| **Fetch** | `Request` → `Response` | network; raw archive write | no |
| **Parse** | `Response` → `Record`s and follow-up `Request`s | none (pure) | yes, versioned |
| **Load** | `Record` → canonical rows | DB writes (single code path) | yes, idempotent |
| **Enrich** | canonical rows → `Record`s | via Loader only | yes, idempotent |
Because parse is pure and responses are archived, `politrack reparse <adapter>` rebuilds data
after a parser fix without re-fetching. Because load is the only writer, invariants (no claim
without evidence, data minimisation, identity rules) live in one place.
## 3. Data model
### 3.1 Principles
* **Claim/evidence**: every user-visible fact is a `claim` with ≥1 `claim_evidence` row pointing
to a `document` (a URL) published by a `source`. Claims from different sources about the same
fact collapse into one claim with several evidence rows ("corroborated").
* **Typed facets**: claims of some types carry a structured facet row (role, vote position,
money flow, event participation, statement, legal proceeding). Facets make analytics and
correlations possible; the claim makes provenance and display uniform.
* **Identifiers instead of ID columns**: entities get any number of external identifiers in one
table (`scheme`, `value`), e.g. `parlament:2339`, `wikidata:Q123`, `ep:197479`,
`bluesky:did:plc:…`, `telegram:HKickl`, `uid:ATU12345678`, `rtr:7846`. New sources add schemes,
not columns.
* **Bitemporal enough**: claims have `occurred_at` (point events) or `valid_from/valid_to`
(states such as mandates); documents have `published_at`; everything has `created_at/updated_at`.
### 3.2 Tables (abridged; full DDL in `backend/migrations`)
```
source(slug PK-ish, name, kind, homepage, license, access, trust, store_fulltext, keep_raw, tos_notes, …)
fetch(id, adapter, source_id, request_fp, url, method, status, sha256, size, content_type,
etag, last_modified, fetched_at, request jsonb, parsed_sha256, parser_version) -- raw archive index
document(id, source_id, url UNIQUE, title, doc_type, lang, published_at, excerpt, full_text,
fetch_id, story_id, meta jsonb)
person(id, slug, full_name, given_name, family_name, prefixes…, birth_date, death_date, gender, photo_url)
organization(id, slug, name, short_name, kind, parent_id, color, …)
identifier(entity_type, entity_id, scheme, value, source_slug, verified) UNIQUE(entity_type, scheme, value)
alias(entity_type, entity_id, alias, alias_norm)
social_account(id, person_id|organization_id, platform, handle, url, verified, source_slug)
claim(id, claim_type, fingerprint UNIQUE, person_id, organization_id, counterpart_person_id,
counterpart_org_id, occurred_at, valid_from, valid_to, summary_de, summary_en, data jsonb,
status, sensitive, review_status, merged_into, created_by, …)
claim_evidence(claim_id, document_id, stance, quote, locator, extracted_by, confidence)
role / vote_event / vote_position / money_flow / event / event_participant / statement /
legal_proceeding -- facets
topic, topic_assignment, story, chunk(text, tsv, embedding vector(1024))
run(id, adapter, started_at, finished_at, status, stats, error), state(adapter, key, value)
```
### 3.3 Claim identity
The *Loader* (not the scraper) computes a claim's fingerprint from **resolved entity ids** and
the semantically identifying fields of its type:
| claim type | identity |
|---|---|
| role | person, body, canonical role, start date |
| vote | vote event key, voter (person or party) |
| donation / money_flow | kind, payer, payee, amount, date, external reference |
| statement | person, URL |
| media_mention | URL, subject |
| meeting / travel | person, anchor (counterpart surname or country), ISO week |
| legal_proceeding, others | explicit key from the adapter |
Using resolved ids means a Wikidata mandate and a Parliament mandate of the same person, body
and start date *are the same claim* by construction. A fuzzy **corroboration** pass handles
near-duplicates (dates off by days, wording differences) by merging claims (`merged_into`).
### 3.4 Entity resolution
Records refer to entities through `PersonRef` / `OrgRef`:
```python
PersonRef(ids={"parlament": "2339"}, name="Selma Yildirim") # strong id
PersonRef(name="Auer", hints={"surname_in_body": "nationalrat", "on": date(2025, 9, 24)})
OrgRef(name="Hypo Tirol Bank AG", kind="company", ids={"uid": "ATU…"})
```
Resolution order: (1) any strong identifier → entity; (2) strategy named in hints
(e.g. *surname among members of body on date*); (3) exact normalised alias; (4) for
organisations: normalised legal name (legal-form suffixes stripped); (5) create — persons are
only created by "authoritative" adapters (`creates_persons=True`), other adapters leave
unresolvable person refs unlinked and count them. Ambiguity never guesses.
## 4. The adapter framework (the nitty-gritty)
### 4.1 SourceSpec – single source of truth for a publisher
```python
@dataclass(frozen=True)
class SourceSpec:
slug: str # "parlament"
name: str
kind: Literal["official", "ngo", "media", "social", "aggregator", "user"]
homepage: str
license: str
access: Literal["api", "bulk", "rss", "scrape", "manual"]
trust: int # default 0–100, editorial
store_fulltext: bool # may we keep article/body text?
keep_raw: bool = True # may we archive raw responses? (False for TDM-reserved)
min_interval: float = 2.0 # seconds between requests to one host
robots_exempt: tuple[str, ...] = () # URL prefixes of documented APIs; justify in tos_notes
tos_notes: str = ""
```
Specs live next to their adapter. `politrack sources sync` (run by `migrate`) upserts them into
the `source` table, so SQL never needs editing to add a source.
### 4.2 Adapter
```python
class Adapter:
spec: ClassVar[SourceSpec]
name: ClassVar[str] # "parlament.items" (a source may have several adapters)
schedule: ClassVar[str | None] # cron (UTC) or None = manual only
queue: ClassVar[str] = "scrape"
parser_version: ClassVar[int] = 1 # bump when parse() output changes → reparse
def start(self, ctx: RunContext) -> Iterable[Request]: ...
def parse(self, ctx: RunContext, response: Response) -> Iterable[Record | Request]: ...
```
* `start()` yields the entry requests (list endpoints, index pages, feeds). It may read
`ctx.state` (cursors) and `ctx.since`.
* `parse()` is **pure**: no DB access, no network. It yields records and follow-up requests
(detail pages). Dispatch on `response.request.kind` (`"list"`, `"detail"`, …).
* Anything that needs the database (e.g. "who sat in the Nationalrat on that day") is expressed
as resolution hints on refs and done by the Loader.
### 4.3 Request / Response
```python
@dataclass(frozen=True)
class Request:
url: str
kind: str = "default" # parse() dispatch key
method: str = "GET"
params: dict | None = None
json: Any = None
headers: dict | None = None
meta: dict = field(default_factory=dict) # passed through to parse (e.g. {"gp": "XXVIII"})
conditional: bool = True # send If-None-Match / If-Modified-Since
skip_unchanged: bool = True # don't reload records if body hash and parser_version unchanged
@dataclass
class Response:
request: Request
status: int
url: str
headers: Mapping[str, str]
body: bytes
fetched_at: datetime
sha256: str
fetch_id: int | None # row in the raw archive index
def text(self) -> str; def json(self) -> Any; def soup(self) -> BeautifulSoup
```
### 4.4 Fetcher (policy enforcement)
For every request the fetcher:
1. checks robots.txt (cached per origin) unless the URL starts with a `robots_exempt` prefix;
2. throttles per host (`max(spec.min_interval, robots crawl-delay)`);
3. sends an honest User-Agent with contact address, and conditional headers from the last fetch;
4. retries 429/5xx with exponential back-off, honouring `Retry-After`;
5. detects TDM reservations (`tdm-reservation` header, `/.well-known/tdmrep.json`) and records
them on the response (`response.tdm_reserved`) – the Loader then drops full text;
6. writes the body to the raw archive (`sha256`-addressed, gzip) **iff** `spec.keep_raw` and no
TDM reservation, and logs a `fetch_log` row in every case.
### 4.5 Records (typed, validated)
```python
Evidence(url, title=None, doc_type="web_page", published_at=None, excerpt=None, full_text=None,
quote=None, locator="", stance="supports", confidence=1.0, source=None)
PersonRecord(ref, full_name, given_name=None, family_name=None, …, aliases=(), social=())
OrgRecord(ref, name, short_name=None, kind="other", parent=None, color=None)
Claim(claim_type, summary_de, evidence, person=None, org=None, counterpart_person=None,
counterpart_org=None, occurred_at=None, valid_from=None, valid_to=None, summary_en=None,
data={}, topics=(), sensitive=False, review=False, key=None, facet=None)
# facets
Role(role, level, body=None, party=None, region=None, start=None, end=None)
VoteEvent(key, chamber, title, voted_on, result, stage=None, item_ref=None, roll_call=False, url=None, …)
VotePosition(event_key, position) # voter = claim.person or claim.org
Money(kind, amount, currency="EUR", payer=None, payee=None, payer_name=None, payee_name=None,
date=None, reference=None, category=None, period=None)
Event(key, kind, title, starts_at, ends_at=None, location=None, country=None, url=None, body=None)
Participation(event_key, role="participant")
Statement(context, url, said_at=None, medium=None, title=None, excerpt=None, platform=None)
Legal(stage, offense=None, authority=None, case_ref=None, stage_date=None)
Document(evidence) # a document without a claim (e.g. a headline);
# enrichers create claims from it later
```
Data minimisation is enforced by *schema*: e.g. `Money` has no address field, so a parser
cannot leak a donor's street address into the database even by accident.
### 4.6 Runner
```python
def run(adapter, mode="fetch", since=None):
ctx = RunContext(adapter, since, state=StateStore(adapter.name), stats=Counter())
queue = deque(adapter.start(ctx)); seen = set()
while queue:
req = queue.popleft()
if fingerprint(req) in seen: continue; seen.add(fingerprint(req))
resp = fetcher.fetch(req) if mode == "fetch" else archive.latest(req)
if resp is None: continue
if req.skip_unchanged and archive.already_parsed(resp, adapter.parser_version):
# records already loaded; still follow links so an interrupted run's children get fetched
queue.extend(r for r in adapter.parse(ctx, resp) if isinstance(r, Request)); continue
with db.transaction(): # one transaction per response
for item in adapter.parse(ctx, resp):
queue.append(item) if isinstance(item, Request) else loader.load(item, ctx)
archive.mark_parsed(resp, adapter.parser_version)
record run row with stats/errors
```
Per-response transactions mean a crash loses one page, never the whole run; errors in one
response are recorded and the run continues (error budget → run status `partial`).
### 4.7 Scheduling & deployment
* Procrastinate (Postgres-backed queue): the worker builds one periodic task per adapter from
`adapter.schedule`/`adapter.queue` – no hand-written task list.
* Queues `scrape`, `social`, `bulk`, `enrich` let heavy bulk jobs (procurement, ad archives) run
on their own worker (even another machine) without blocking light ones.
* One Docker image for api/worker/mcp/migrate; third-party adapter packages register via the
Python entry-point group `politrack.adapters`, so a separate deployment can add sources
without forking.
### 4.8 Enrichers
Same Loader, different input: an `Enricher` has `run(ctx) -> Iterable[Record]` over database
rows (documents not yet linked, claims not yet tagged…). Built-ins: mention linker,
event extractor (trips/meetings), legal-matter detector (→ review queue), topic tagger,
corroborator (merge near-duplicates, cluster news stories), say-vs-vote matcher, chunker +
embedder.
## 5. Serving
* **API**: FastAPI with Pydantic response models (OpenAPI is the contract). Standard filters
everywhere: `person`, `party`, `types`, `topics`, `since`, `until`, `exclude_sources`,
`min_trust`, `limit/offset`.
* **Web**: Next.js App Router + next-intl (de at `/`, en at `/en`); TypeScript types generated
from OpenAPI (`politrack openapi --ts`); preferences in a cookie so server components can
filter; a **renderer registry** maps claim types to components (adding a claim type = one
component).
* **Agents**: MCP tools mirroring the API; read-only SQL role; `COMMENT ON` everything;
`claim_with_evidence` view; `chunk` table for RAG with citations.
## 6. Testing strategy
| Level | What | How |
|---|---|---|
| Parser golden tests | each adapter's `parse()` | real responses saved as fixtures (`tests/fixtures/<adapter>/…`), expected records as JSON next to them; `UPDATE_GOLDEN=1` regenerates |
| Unit tests | text normalisation, resolvers, identity | plain pytest |
| Loader/integration | load records into an ephemeral Postgres, assert rows | `pytest -m integration` with `POLITRACK_TEST_DATABASE_URL` |
| API smoke | every endpoint against a seeded DB | FastAPI TestClient |
| Frontend | type check + build | `npm run typecheck && npm run build` |
## 7. Legal & ethical guard-rails in code
* Fetcher enforces robots/TDM/rate limits; `keep_raw`/`store_fulltext` per source.
* Record schemas exclude data we must not keep (addresses, private contact data).
* `sensitive=True` claims render the presumption-of-innocence notice unless stage is an
undisputed final conviction; auto-detected legal matters require `review`.
* Messengers: only *public broadcast channels* (e.g. public Telegram channels' web preview);
never private groups, never platforms whose terms forbid automated access (WhatsApp, X without
API, Facebook/Instagram).
## 8. Operations
* Services: `db` (Postgres 16 + pgvector), `migrate` (one-shot), `api`, `web`, `mcp`,
`worker-scrape`, `worker-bulk`, `worker-enrich`, optional `ollama`.
* Volumes: Postgres data, raw archive (`/data/raw`), Ollama models.
* Observability: `run` table, `politrack status`, `/api/v1/admin/runs` (token), structured logs.
* Backups: `pg_dump` + raw archive rsync (raw archive is optional to back up – it is a cache
that can be re-fetched, but it is also the audit trail).
@@ -0,0 +1,92 @@
# Clean-room design vs. first implementation – decisions
The first implementation (v1) grew iteratively. The clean-room design
([01-design-from-scratch.md](01-design-from-scratch.md)) was written without looking at it.
This document compares them area by area and records the decision taken for v2 (the current
code). Format: lightweight ADRs.
## Summary
| # | Area | v1 (first implementation) | Clean-room design | **Decision (v2)** |
|---|---|---|---|---|
| 1 | Pipeline shape | each scraper fetches, parses and writes in one method | fetch → archive → pure parse → loader | **Clean-room**. Bugs found during v1 testing (vote parser, roll calls, offence regex) required resetting state and re-fetching; with an archive they are a `reparse`. |
| 2 | Writing to the DB | scrapers call `store.record_claim(...)` with hand-made fingerprint keys | typed records; Loader computes identity from resolved entity ids | **Clean-room**, keeping v1's battle-tested upsert SQL inside the Loader. |
| 3 | Source definition | SQL seed + scraper class + worker schedule + YAML, four places | `SourceSpec` in code, synced to DB; schedule on the adapter | **Clean-room**. One module per source. |
| 4 | External ids | columns `parlament_pad`, `wikidata_qid` | `identifier(scheme, value)` table | **Clean-room**. Needed anyway for EP, Bluesky DIDs, Telegram, company UIDs, RTR ids. |
| 5 | Entity references | scrapers look persons up themselves (SQL in scrapers) | `PersonRef`/`OrgRef` with ids/name/hints, resolved by the Loader | **Clean-room**. Makes parsers pure and testable. |
| 6 | Change detection | per-scraper ad-hoc signatures in `scrape_state` | generic: body hash + parser version per request fingerprint; conditional GET | **Clean-room**; v1's cursor state store kept for real cursors (dates, ids). |
| 7 | Policy enforcement | `PoliteClient` (robots, TDM, throttle, retries) + per-call `documented_api=True` | same, but exemptions declared on `SourceSpec.robots_exempt` | **Merge**: v1 client code + declarative exemptions. |
| 8 | Corroboration | post-hoc merge engine (roles, stories, events) | identity by construction + fuzzy merge pass | **Both**: identity by construction removes most duplicates; v1's merge engine handles near-duplicates. |
| 9 | Activity status | `person_status` view with verified flag and term windows | not specified in detail | **v1** (proven on real data: Van der Bellen/Mikl-Leitner cases). |
| 10 | Legal matters | review queue, innocence-notice rule, official immunity data | same principles | **v1** implementation, moved behind the Loader (`review=True`). |
| 11 | Scheduling | Procrastinate, tasks written by hand | Procrastinate, tasks generated from adapters; extra `bulk`/`social` queues | **Clean-room**. |
| 12 | Enrichment | functions writing claims directly | enrichers emitting records to the Loader | **Clean-room**. |
| 13 | API contract | handwritten dict responses; hand-maintained TS types | Pydantic response models, generated TS types | **Clean-room**, generated with a small built-in generator (`politrack openapi --ts`) so it works without npm. |
| 14 | Frontend claim rendering | one generic `ClaimList` with `if` branches | renderer registry per claim type | **Clean-room** (generic fallback + specific renderers). |
| 15 | Tests | inline-string unit tests | golden tests from real fixtures + unit + integration + API smoke | **Clean-room**, keeping v1's unit tests. |
| 16 | Data minimisation | sanitiser function in the donations scraper | enforced by record schemas | **Clean-room** + keep the sanitiser (defence in depth). |
| 17 | Storage of raw data | none | content-addressed raw archive, per-source `keep_raw` | **Clean-room**. |
| 18 | Plugins | none | entry points `politrack.adapters` | **Clean-room** (cheap to add, enables separate deployments). |
| 19 | Stack choices (Postgres+pgvector, FastAPI, Next.js+next-intl, MCP, Docker Compose) | ✓ | ✓ | **Same in both** – confirmed. |
| 20 | Scrapy vs. own framework | own | own (Scrapy considered) | **Own**. See ADR-20. |
## Selected ADRs
### ADR-1: Archive raw responses and make parsers pure
**Context.** In v1 three parser bugs were found after a full run; fixing them required
deleting state, rows and refetching ~1,000 pages (and politely waiting).
**Decision.** Every fetch is logged; bodies are archived (gzip, sha256-addressed) unless the
source forbids it (`keep_raw=False`, TDM reservation). Parsers are pure functions of the
response. `politrack reparse <adapter>` replays the archive.
**Consequences.** Disk usage (Parliament JSON + protocols ≈ a few GB/year; bulk sources more –
retention configurable). Parsers become trivially unit-testable with real fixtures.
### ADR-2: The Loader is the only writer
**Decision.** Adapters and enrichers yield typed records; `Loader.load()` resolves references,
computes identity, upserts documents/claims/evidence/facets, assigns topics and refreshes
status. Invariants (no claim without evidence, full text only where licensed, review gating)
are enforced there.
### ADR-4: `identifier` table
**Decision.** `identifier(entity_type, entity_id, scheme, value)`; schemes are free-form
strings documented in `docs/data-model.md` (`parlament`, `wikidata`, `ep`, `bluesky`,
`mastodon`, `telegram`, `youtube`, `uid`, `firmenbuch`, `rtr`, `eu_transparency`, …).
### ADR-6: Generic change detection
**Decision.** The fetcher stores ETag/Last-Modified per request fingerprint and sends
conditional requests; the runner skips `parse()` when the body hash and parser version equal the
last successful parse. Adapters no longer keep "seen" dictionaries.
### ADR-13: Typed API, generated TS types without npm
**Context.** The development network blocks npm; contributors should still be able to
regenerate types. **Decision.** `politrack openapi` writes `openapi.json` and
`web/src/lib/api-types.ts` from the FastAPI app using a small generator for the subset of
JSON-Schema FastAPI emits.
### ADR-20: No Scrapy
**Context.** Scrapy offers robots middleware, throttling, HTTP cache and pipelines.
**Decision.** Keep a small own framework (~600 lines): our sources are mostly APIs and bulk
files, we need DB-integrated change detection, Procrastinate scheduling, TDM handling and pure
parsers with typed records. Scrapy's Twisted runtime and project layout would add more
complexity than it removes. Revisit if we ever crawl large websites.
### ADR-21: Messaging platforms
**Decision.** Public Telegram *channels* via their public web preview (`t.me/s/<channel>`,
no robots restrictions, read-only, low frequency) and only for handles from verifiable
sources (Wikidata P3789, official websites). WhatsApp Channels and Signal are not collected:
WhatsApp offers no public read access and its terms prohibit automated collection; Signal has
no public channels and groups are private by design.
### ADR-22: X only through the official API, opt-in
**Context.** Many Austrian politicians are most active on X; proxies such as Nitter work by violating
X's terms. **Decision.** `social.x` uses the official API v2 with the operator's bearer token and does
nothing without one. It keeps no raw archive and no full text (`keep_raw=False`, `store_fulltext=False`).
The Developer Agreement requires removing posts deleted on X; until a deletion sync exists the adapter is
documented as not production-ready.
### ADR-23: Organisation-owned accounts
**Context.** Party and institution channels (MFG on Telegram, NEOS and the Federal President on Bluesky)
belong to organisations, not people. **Decision.** `social_account` and statements may belong to an
organisation (`statement.person_id` NULL, `claim.organization_id` set); the social feed shows them, and
party pages include the party's own channels. Curated accounts live in `config/social_accounts.yaml`
(imported by `config.social_accounts`), each with the official page that links it.