Politrack: track Austrian politicians from public sources

Initial import of Politrack, a self-hosted service that tracks Austrian
politicians from federal to municipal level - roles, votes, money flows,
statements, meetings, social media and legal proceedings - with every claim
linked to its sources.

- backend: FastAPI + Postgres/pgvector; adapter framework (fetch -> parse ->
  load -> enrich) with robots.txt/TDM-aware fetching, a raw-response archive
  for reparsing, and a claim/evidence model with corroboration
- sources: Parliament, Wikidata, EU Parliament/Commission, Council of
  Ministers, Rechnungshof party reports and donations, procurement, RTR media
  transparency, political ad libraries, WKStA, news feeds, Wikipedia edits,
  Bluesky, Mastodon, public Telegram channels, YouTube and X (official API)
- web: Next.js frontend in German and English with typed API client
- mcp: MCP server for AI agents and RAG
- docs: architecture, data model, sources, legal notes, Unraid deployment and
  contributor guides; Docker Compose setup for Unraid

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
simonandClaude Opus 5.5 committed 2026-10-28 20:02:01 +01:00
commit 583ccba31c
187 files changed
+39890

No files matched your search

+325
View File
@@ -0,0 +1,325 @@
# Politrack – clean-room design
> Written as if no code existed. Inputs were only the requirements: track what Austrian
> politicians (federal → municipal) do, from many heterogeneous public sources, legally, with
> every fact backed by links; user-selectable trust & topics; person/party overviews; calendar
> and timeline; multilingual (de default, en); agent/RAG access; runs on an Unraid homelab in
> Docker; scrapers deployable separately; easy for contributors to extend.
>
> [02-comparison-and-decisions.md](02-comparison-and-decisions.md) compares this design with
> the first implementation and records which decisions won.
## 1. Quality goals (in priority order)
1. **Trustworthiness** – no fact without provenance; never present something as more certain
than its sources; sensitive facts (criminal matters) are gated. Precision beats recall.
2. **Legal compliance by construction** – the framework, not the individual scraper author,
enforces robots.txt, TDM reservations, rate limits, storage limits and data minimisation.
3. **Reproducibility** – any derived row can be explained ("which source, which response,
which parser version") and regenerated from archived raw data after a parser fix, without
hitting the source again.
4. **Extensibility** – adding a source must touch exactly one module (+ a fixture-based test
and a docs line). No edits to SQL seeds, schedulers or registries elsewhere.
5. **Operability on a homelab** – few moving parts (one database, one image), self-healing
schedules, everything inspectable with SQL.
## 2. System context
```
┌───────────── public sources (APIs, open data, feeds, sitemaps, archives) ─────────────┐
▼ │
┌────────────────┐ Request ┌──────────┐ Response ┌──────────┐ Records ┌────────┐ │
│ Adapter.start()│──────────▶ │ Fetcher │─────────▶ │ Adapter. │───────▶ │ Loader │──▶ Postgres│
│ (per source) │◀────────── │ (policy, │ archive │ parse() │ FollowUp│ │ │
└────────────────┘ FollowUp │ archive) │──▶ Raw └──────────┘ └────────┘ │
└──────────┘ Archive (blobs + fetch_log) ▲ │
│ Records │
Enrichers (linking, events, legal, topics, ─────┘ │
corroboration, embeddings) read Postgres │
│
Serving: FastAPI (typed JSON, OpenAPI) ─▶ Next.js web (de/en) MCP server ─▶ AI agents │
read-only SQL role ─▶ notebooks/BI/agents pgvector chunks ─▶ RAG │
```
Four strictly separated stages:
| Stage | Input → output | Side effects | Determinism |
|---|---|---|---|
| **Fetch** | `Request` → `Response` | network; raw archive write | no |
| **Parse** | `Response` → `Record`s and follow-up `Request`s | none (pure) | yes, versioned |
| **Load** | `Record` → canonical rows | DB writes (single code path) | yes, idempotent |
| **Enrich** | canonical rows → `Record`s | via Loader only | yes, idempotent |
Because parse is pure and responses are archived, `politrack reparse <adapter>` rebuilds data
after a parser fix without re-fetching. Because load is the only writer, invariants (no claim
without evidence, data minimisation, identity rules) live in one place.
## 3. Data model
### 3.1 Principles
* **Claim/evidence**: every user-visible fact is a `claim` with ≥1 `claim_evidence` row pointing
to a `document` (a URL) published by a `source`. Claims from different sources about the same
fact collapse into one claim with several evidence rows ("corroborated").
* **Typed facets**: claims of some types carry a structured facet row (role, vote position,
money flow, event participation, statement, legal proceeding). Facets make analytics and
correlations possible; the claim makes provenance and display uniform.
* **Identifiers instead of ID columns**: entities get any number of external identifiers in one
table (`scheme`, `value`), e.g. `parlament:2339`, `wikidata:Q123`, `ep:197479`,
`bluesky:did:plc:…`, `telegram:HKickl`, `uid:ATU12345678`, `rtr:7846`. New sources add schemes,
not columns.
* **Bitemporal enough**: claims have `occurred_at` (point events) or `valid_from/valid_to`
(states such as mandates); documents have `published_at`; everything has `created_at/updated_at`.
### 3.2 Tables (abridged; full DDL in `backend/migrations`)
```
source(slug PK-ish, name, kind, homepage, license, access, trust, store_fulltext, keep_raw, tos_notes, …)
fetch(id, adapter, source_id, request_fp, url, method, status, sha256, size, content_type,
etag, last_modified, fetched_at, request jsonb, parsed_sha256, parser_version) -- raw archive index
document(id, source_id, url UNIQUE, title, doc_type, lang, published_at, excerpt, full_text,
fetch_id, story_id, meta jsonb)
person(id, slug, full_name, given_name, family_name, prefixes…, birth_date, death_date, gender, photo_url)
organization(id, slug, name, short_name, kind, parent_id, color, …)
identifier(entity_type, entity_id, scheme, value, source_slug, verified) UNIQUE(entity_type, scheme, value)
alias(entity_type, entity_id, alias, alias_norm)
social_account(id, person_id|organization_id, platform, handle, url, verified, source_slug)
claim(id, claim_type, fingerprint UNIQUE, person_id, organization_id, counterpart_person_id,
counterpart_org_id, occurred_at, valid_from, valid_to, summary_de, summary_en, data jsonb,
status, sensitive, review_status, merged_into, created_by, …)
claim_evidence(claim_id, document_id, stance, quote, locator, extracted_by, confidence)
role / vote_event / vote_position / money_flow / event / event_participant / statement /
legal_proceeding -- facets
topic, topic_assignment, story, chunk(text, tsv, embedding vector(1024))
run(id, adapter, started_at, finished_at, status, stats, error), state(adapter, key, value)
```
### 3.3 Claim identity
The *Loader* (not the scraper) computes a claim's fingerprint from **resolved entity ids** and
the semantically identifying fields of its type:
| claim type | identity |
|---|---|
| role | person, body, canonical role, start date |
| vote | vote event key, voter (person or party) |
| donation / money_flow | kind, payer, payee, amount, date, external reference |
| statement | person, URL |
| media_mention | URL, subject |
| meeting / travel | person, anchor (counterpart surname or country), ISO week |
| legal_proceeding, others | explicit key from the adapter |
Using resolved ids means a Wikidata mandate and a Parliament mandate of the same person, body
and start date *are the same claim* by construction. A fuzzy **corroboration** pass handles
near-duplicates (dates off by days, wording differences) by merging claims (`merged_into`).
### 3.4 Entity resolution
Records refer to entities through `PersonRef` / `OrgRef`:
```python
PersonRef(ids={"parlament": "2339"}, name="Selma Yildirim") # strong id
PersonRef(name="Auer", hints={"surname_in_body": "nationalrat", "on": date(2025, 9, 24)})
OrgRef(name="Hypo Tirol Bank AG", kind="company", ids={"uid": "ATU…"})
```
Resolution order: (1) any strong identifier → entity; (2) strategy named in hints
(e.g. *surname among members of body on date*); (3) exact normalised alias; (4) for
organisations: normalised legal name (legal-form suffixes stripped); (5) create — persons are
only created by "authoritative" adapters (`creates_persons=True`), other adapters leave
unresolvable person refs unlinked and count them. Ambiguity never guesses.
## 4. The adapter framework (the nitty-gritty)
### 4.1 SourceSpec – single source of truth for a publisher
```python
@dataclass(frozen=True)
class SourceSpec:
slug: str # "parlament"
name: str
kind: Literal["official", "ngo", "media", "social", "aggregator", "user"]
homepage: str
license: str
access: Literal["api", "bulk", "rss", "scrape", "manual"]
trust: int # default 0–100, editorial
store_fulltext: bool # may we keep article/body text?
keep_raw: bool = True # may we archive raw responses? (False for TDM-reserved)
min_interval: float = 2.0 # seconds between requests to one host
robots_exempt: tuple[str, ...] = () # URL prefixes of documented APIs; justify in tos_notes
tos_notes: str = ""
```
Specs live next to their adapter. `politrack sources sync` (run by `migrate`) upserts them into
the `source` table, so SQL never needs editing to add a source.
### 4.2 Adapter
```python
class Adapter:
spec: ClassVar[SourceSpec]
name: ClassVar[str] # "parlament.items" (a source may have several adapters)
schedule: ClassVar[str | None] # cron (UTC) or None = manual only
queue: ClassVar[str] = "scrape"
parser_version: ClassVar[int] = 1 # bump when parse() output changes → reparse
def start(self, ctx: RunContext) -> Iterable[Request]: ...
def parse(self, ctx: RunContext, response: Response) -> Iterable[Record | Request]: ...
```
* `start()` yields the entry requests (list endpoints, index pages, feeds). It may read
`ctx.state` (cursors) and `ctx.since`.
* `parse()` is **pure**: no DB access, no network. It yields records and follow-up requests
(detail pages). Dispatch on `response.request.kind` (`"list"`, `"detail"`, …).
* Anything that needs the database (e.g. "who sat in the Nationalrat on that day") is expressed
as resolution hints on refs and done by the Loader.
### 4.3 Request / Response
```python
@dataclass(frozen=True)
class Request:
url: str
kind: str = "default" # parse() dispatch key
method: str = "GET"
params: dict | None = None
json: Any = None
headers: dict | None = None
meta: dict = field(default_factory=dict) # passed through to parse (e.g. {"gp": "XXVIII"})
conditional: bool = True # send If-None-Match / If-Modified-Since
skip_unchanged: bool = True # don't reload records if body hash and parser_version unchanged
@dataclass
class Response:
request: Request
status: int
url: str
headers: Mapping[str, str]
body: bytes
fetched_at: datetime
sha256: str
fetch_id: int | None # row in the raw archive index
def text(self) -> str; def json(self) -> Any; def soup(self) -> BeautifulSoup
```
### 4.4 Fetcher (policy enforcement)
For every request the fetcher:
1. checks robots.txt (cached per origin) unless the URL starts with a `robots_exempt` prefix;
2. throttles per host (`max(spec.min_interval, robots crawl-delay)`);
3. sends an honest User-Agent with contact address, and conditional headers from the last fetch;
4. retries 429/5xx with exponential back-off, honouring `Retry-After`;
5. detects TDM reservations (`tdm-reservation` header, `/.well-known/tdmrep.json`) and records
them on the response (`response.tdm_reserved`) – the Loader then drops full text;
6. writes the body to the raw archive (`sha256`-addressed, gzip) **iff** `spec.keep_raw` and no
TDM reservation, and logs a `fetch_log` row in every case.
### 4.5 Records (typed, validated)
```python
Evidence(url, title=None, doc_type="web_page", published_at=None, excerpt=None, full_text=None,
quote=None, locator="", stance="supports", confidence=1.0, source=None)
PersonRecord(ref, full_name, given_name=None, family_name=None, …, aliases=(), social=())
OrgRecord(ref, name, short_name=None, kind="other", parent=None, color=None)
Claim(claim_type, summary_de, evidence, person=None, org=None, counterpart_person=None,
counterpart_org=None, occurred_at=None, valid_from=None, valid_to=None, summary_en=None,
data={}, topics=(), sensitive=False, review=False, key=None, facet=None)
# facets
Role(role, level, body=None, party=None, region=None, start=None, end=None)
VoteEvent(key, chamber, title, voted_on, result, stage=None, item_ref=None, roll_call=False, url=None, …)
VotePosition(event_key, position) # voter = claim.person or claim.org
Money(kind, amount, currency="EUR", payer=None, payee=None, payer_name=None, payee_name=None,
date=None, reference=None, category=None, period=None)
Event(key, kind, title, starts_at, ends_at=None, location=None, country=None, url=None, body=None)
Participation(event_key, role="participant")
Statement(context, url, said_at=None, medium=None, title=None, excerpt=None, platform=None)
Legal(stage, offense=None, authority=None, case_ref=None, stage_date=None)
Document(evidence) # a document without a claim (e.g. a headline);
# enrichers create claims from it later
```
Data minimisation is enforced by *schema*: e.g. `Money` has no address field, so a parser
cannot leak a donor's street address into the database even by accident.
### 4.6 Runner
```python
def run(adapter, mode="fetch", since=None):
ctx = RunContext(adapter, since, state=StateStore(adapter.name), stats=Counter())
queue = deque(adapter.start(ctx)); seen = set()
while queue:
req = queue.popleft()
if fingerprint(req) in seen: continue; seen.add(fingerprint(req))
resp = fetcher.fetch(req) if mode == "fetch" else archive.latest(req)
if resp is None: continue
if req.skip_unchanged and archive.already_parsed(resp, adapter.parser_version):
# records already loaded; still follow links so an interrupted run's children get fetched
queue.extend(r for r in adapter.parse(ctx, resp) if isinstance(r, Request)); continue
with db.transaction(): # one transaction per response
for item in adapter.parse(ctx, resp):
queue.append(item) if isinstance(item, Request) else loader.load(item, ctx)
archive.mark_parsed(resp, adapter.parser_version)
record run row with stats/errors
```
Per-response transactions mean a crash loses one page, never the whole run; errors in one
response are recorded and the run continues (error budget → run status `partial`).
### 4.7 Scheduling & deployment
* Procrastinate (Postgres-backed queue): the worker builds one periodic task per adapter from
`adapter.schedule`/`adapter.queue` – no hand-written task list.
* Queues `scrape`, `social`, `bulk`, `enrich` let heavy bulk jobs (procurement, ad archives) run
on their own worker (even another machine) without blocking light ones.
* One Docker image for api/worker/mcp/migrate; third-party adapter packages register via the
Python entry-point group `politrack.adapters`, so a separate deployment can add sources
without forking.
### 4.8 Enrichers
Same Loader, different input: an `Enricher` has `run(ctx) -> Iterable[Record]` over database
rows (documents not yet linked, claims not yet tagged…). Built-ins: mention linker,
event extractor (trips/meetings), legal-matter detector (→ review queue), topic tagger,
corroborator (merge near-duplicates, cluster news stories), say-vs-vote matcher, chunker +
embedder.
## 5. Serving
* **API**: FastAPI with Pydantic response models (OpenAPI is the contract). Standard filters
everywhere: `person`, `party`, `types`, `topics`, `since`, `until`, `exclude_sources`,
`min_trust`, `limit/offset`.
* **Web**: Next.js App Router + next-intl (de at `/`, en at `/en`); TypeScript types generated
from OpenAPI (`politrack openapi --ts`); preferences in a cookie so server components can
filter; a **renderer registry** maps claim types to components (adding a claim type = one
component).
* **Agents**: MCP tools mirroring the API; read-only SQL role; `COMMENT ON` everything;
`claim_with_evidence` view; `chunk` table for RAG with citations.
## 6. Testing strategy
| Level | What | How |
|---|---|---|
| Parser golden tests | each adapter's `parse()` | real responses saved as fixtures (`tests/fixtures/<adapter>/…`), expected records as JSON next to them; `UPDATE_GOLDEN=1` regenerates |
| Unit tests | text normalisation, resolvers, identity | plain pytest |
| Loader/integration | load records into an ephemeral Postgres, assert rows | `pytest -m integration` with `POLITRACK_TEST_DATABASE_URL` |
| API smoke | every endpoint against a seeded DB | FastAPI TestClient |
| Frontend | type check + build | `npm run typecheck && npm run build` |
## 7. Legal & ethical guard-rails in code
* Fetcher enforces robots/TDM/rate limits; `keep_raw`/`store_fulltext` per source.
* Record schemas exclude data we must not keep (addresses, private contact data).
* `sensitive=True` claims render the presumption-of-innocence notice unless stage is an
undisputed final conviction; auto-detected legal matters require `review`.
* Messengers: only *public broadcast channels* (e.g. public Telegram channels' web preview);
never private groups, never platforms whose terms forbid automated access (WhatsApp, X without
API, Facebook/Instagram).
## 8. Operations
* Services: `db` (Postgres 16 + pgvector), `migrate` (one-shot), `api`, `web`, `mcp`,
`worker-scrape`, `worker-bulk`, `worker-enrich`, optional `ollama`.
* Volumes: Postgres data, raw archive (`/data/raw`), Ollama models.
* Observability: `run` table, `politrack status`, `/api/v1/admin/runs` (token), structured logs.
* Backups: `pg_dump` + raw archive rsync (raw archive is optional to back up – it is a cache
that can be re-fetched, but it is also the audit trail).