Initial import of Politrack, a self-hosted service that tracks Austrian politicians from federal to municipal level - roles, votes, money flows, statements, meetings, social media and legal proceedings - with every claim linked to its sources. - backend: FastAPI + Postgres/pgvector; adapter framework (fetch -> parse -> load -> enrich) with robots.txt/TDM-aware fetching, a raw-response archive for reparsing, and a claim/evidence model with corroboration - sources: Parliament, Wikidata, EU Parliament/Commission, Council of Ministers, Rechnungshof party reports and donations, procurement, RTR media transparency, political ad libraries, WKStA, news feeds, Wikipedia edits, Bluesky, Mastodon, public Telegram channels, YouTube and X (official API) - web: Next.js frontend in German and English with typed API client - mcp: MCP server for AI agents and RAG - docs: architecture, data model, sources, legal notes, Unraid deployment and contributor guides; Docker Compose setup for Unraid Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
326 lines
18 KiB
Markdown
326 lines
18 KiB
Markdown
# Politrack – clean-room design
|
||
|
||
> Written as if no code existed. Inputs were only the requirements: track what Austrian
|
||
> politicians (federal → municipal) do, from many heterogeneous public sources, legally, with
|
||
> every fact backed by links; user-selectable trust & topics; person/party overviews; calendar
|
||
> and timeline; multilingual (de default, en); agent/RAG access; runs on an Unraid homelab in
|
||
> Docker; scrapers deployable separately; easy for contributors to extend.
|
||
>
|
||
> [02-comparison-and-decisions.md](02-comparison-and-decisions.md) compares this design with
|
||
> the first implementation and records which decisions won.
|
||
|
||
## 1. Quality goals (in priority order)
|
||
|
||
1. **Trustworthiness** – no fact without provenance; never present something as more certain
|
||
than its sources; sensitive facts (criminal matters) are gated. Precision beats recall.
|
||
2. **Legal compliance by construction** – the framework, not the individual scraper author,
|
||
enforces robots.txt, TDM reservations, rate limits, storage limits and data minimisation.
|
||
3. **Reproducibility** – any derived row can be explained ("which source, which response,
|
||
which parser version") and regenerated from archived raw data after a parser fix, without
|
||
hitting the source again.
|
||
4. **Extensibility** – adding a source must touch exactly one module (+ a fixture-based test
|
||
and a docs line). No edits to SQL seeds, schedulers or registries elsewhere.
|
||
5. **Operability on a homelab** – few moving parts (one database, one image), self-healing
|
||
schedules, everything inspectable with SQL.
|
||
|
||
## 2. System context
|
||
|
||
```
|
||
┌───────────── public sources (APIs, open data, feeds, sitemaps, archives) ─────────────┐
|
||
▼ │
|
||
┌────────────────┐ Request ┌──────────┐ Response ┌──────────┐ Records ┌────────┐ │
|
||
│ Adapter.start()│──────────▶ │ Fetcher │─────────▶ │ Adapter. │───────▶ │ Loader │──▶ Postgres│
|
||
│ (per source) │◀────────── │ (policy, │ archive │ parse() │ FollowUp│ │ │
|
||
└────────────────┘ FollowUp │ archive) │──▶ Raw └──────────┘ └────────┘ │
|
||
└──────────┘ Archive (blobs + fetch_log) ▲ │
|
||
│ Records │
|
||
Enrichers (linking, events, legal, topics, ─────┘ │
|
||
corroboration, embeddings) read Postgres │
|
||
│
|
||
Serving: FastAPI (typed JSON, OpenAPI) ─▶ Next.js web (de/en) MCP server ─▶ AI agents │
|
||
read-only SQL role ─▶ notebooks/BI/agents pgvector chunks ─▶ RAG │
|
||
```
|
||
|
||
Four strictly separated stages:
|
||
|
||
| Stage | Input → output | Side effects | Determinism |
|
||
|---|---|---|---|
|
||
| **Fetch** | `Request` → `Response` | network; raw archive write | no |
|
||
| **Parse** | `Response` → `Record`s and follow-up `Request`s | none (pure) | yes, versioned |
|
||
| **Load** | `Record` → canonical rows | DB writes (single code path) | yes, idempotent |
|
||
| **Enrich** | canonical rows → `Record`s | via Loader only | yes, idempotent |
|
||
|
||
Because parse is pure and responses are archived, `politrack reparse <adapter>` rebuilds data
|
||
after a parser fix without re-fetching. Because load is the only writer, invariants (no claim
|
||
without evidence, data minimisation, identity rules) live in one place.
|
||
|
||
## 3. Data model
|
||
|
||
### 3.1 Principles
|
||
|
||
* **Claim/evidence**: every user-visible fact is a `claim` with ≥1 `claim_evidence` row pointing
|
||
to a `document` (a URL) published by a `source`. Claims from different sources about the same
|
||
fact collapse into one claim with several evidence rows ("corroborated").
|
||
* **Typed facets**: claims of some types carry a structured facet row (role, vote position,
|
||
money flow, event participation, statement, legal proceeding). Facets make analytics and
|
||
correlations possible; the claim makes provenance and display uniform.
|
||
* **Identifiers instead of ID columns**: entities get any number of external identifiers in one
|
||
table (`scheme`, `value`), e.g. `parlament:2339`, `wikidata:Q123`, `ep:197479`,
|
||
`bluesky:did:plc:…`, `telegram:HKickl`, `uid:ATU12345678`, `rtr:7846`. New sources add schemes,
|
||
not columns.
|
||
* **Bitemporal enough**: claims have `occurred_at` (point events) or `valid_from/valid_to`
|
||
(states such as mandates); documents have `published_at`; everything has `created_at/updated_at`.
|
||
|
||
### 3.2 Tables (abridged; full DDL in `backend/migrations`)
|
||
|
||
```
|
||
source(slug PK-ish, name, kind, homepage, license, access, trust, store_fulltext, keep_raw, tos_notes, …)
|
||
fetch(id, adapter, source_id, request_fp, url, method, status, sha256, size, content_type,
|
||
etag, last_modified, fetched_at, request jsonb, parsed_sha256, parser_version) -- raw archive index
|
||
document(id, source_id, url UNIQUE, title, doc_type, lang, published_at, excerpt, full_text,
|
||
fetch_id, story_id, meta jsonb)
|
||
person(id, slug, full_name, given_name, family_name, prefixes…, birth_date, death_date, gender, photo_url)
|
||
organization(id, slug, name, short_name, kind, parent_id, color, …)
|
||
identifier(entity_type, entity_id, scheme, value, source_slug, verified) UNIQUE(entity_type, scheme, value)
|
||
alias(entity_type, entity_id, alias, alias_norm)
|
||
social_account(id, person_id|organization_id, platform, handle, url, verified, source_slug)
|
||
claim(id, claim_type, fingerprint UNIQUE, person_id, organization_id, counterpart_person_id,
|
||
counterpart_org_id, occurred_at, valid_from, valid_to, summary_de, summary_en, data jsonb,
|
||
status, sensitive, review_status, merged_into, created_by, …)
|
||
claim_evidence(claim_id, document_id, stance, quote, locator, extracted_by, confidence)
|
||
role / vote_event / vote_position / money_flow / event / event_participant / statement /
|
||
legal_proceeding -- facets
|
||
topic, topic_assignment, story, chunk(text, tsv, embedding vector(1024))
|
||
run(id, adapter, started_at, finished_at, status, stats, error), state(adapter, key, value)
|
||
```
|
||
|
||
### 3.3 Claim identity
|
||
|
||
The *Loader* (not the scraper) computes a claim's fingerprint from **resolved entity ids** and
|
||
the semantically identifying fields of its type:
|
||
|
||
| claim type | identity |
|
||
|---|---|
|
||
| role | person, body, canonical role, start date |
|
||
| vote | vote event key, voter (person or party) |
|
||
| donation / money_flow | kind, payer, payee, amount, date, external reference |
|
||
| statement | person, URL |
|
||
| media_mention | URL, subject |
|
||
| meeting / travel | person, anchor (counterpart surname or country), ISO week |
|
||
| legal_proceeding, others | explicit key from the adapter |
|
||
|
||
Using resolved ids means a Wikidata mandate and a Parliament mandate of the same person, body
|
||
and start date *are the same claim* by construction. A fuzzy **corroboration** pass handles
|
||
near-duplicates (dates off by days, wording differences) by merging claims (`merged_into`).
|
||
|
||
### 3.4 Entity resolution
|
||
|
||
Records refer to entities through `PersonRef` / `OrgRef`:
|
||
|
||
```python
|
||
PersonRef(ids={"parlament": "2339"}, name="Selma Yildirim") # strong id
|
||
PersonRef(name="Auer", hints={"surname_in_body": "nationalrat", "on": date(2025, 9, 24)})
|
||
OrgRef(name="Hypo Tirol Bank AG", kind="company", ids={"uid": "ATU…"})
|
||
```
|
||
|
||
Resolution order: (1) any strong identifier → entity; (2) strategy named in hints
|
||
(e.g. *surname among members of body on date*); (3) exact normalised alias; (4) for
|
||
organisations: normalised legal name (legal-form suffixes stripped); (5) create — persons are
|
||
only created by "authoritative" adapters (`creates_persons=True`), other adapters leave
|
||
unresolvable person refs unlinked and count them. Ambiguity never guesses.
|
||
|
||
## 4. The adapter framework (the nitty-gritty)
|
||
|
||
### 4.1 SourceSpec – single source of truth for a publisher
|
||
|
||
```python
|
||
@dataclass(frozen=True)
|
||
class SourceSpec:
|
||
slug: str # "parlament"
|
||
name: str
|
||
kind: Literal["official", "ngo", "media", "social", "aggregator", "user"]
|
||
homepage: str
|
||
license: str
|
||
access: Literal["api", "bulk", "rss", "scrape", "manual"]
|
||
trust: int # default 0–100, editorial
|
||
store_fulltext: bool # may we keep article/body text?
|
||
keep_raw: bool = True # may we archive raw responses? (False for TDM-reserved)
|
||
min_interval: float = 2.0 # seconds between requests to one host
|
||
robots_exempt: tuple[str, ...] = () # URL prefixes of documented APIs; justify in tos_notes
|
||
tos_notes: str = ""
|
||
```
|
||
|
||
Specs live next to their adapter. `politrack sources sync` (run by `migrate`) upserts them into
|
||
the `source` table, so SQL never needs editing to add a source.
|
||
|
||
### 4.2 Adapter
|
||
|
||
```python
|
||
class Adapter:
|
||
spec: ClassVar[SourceSpec]
|
||
name: ClassVar[str] # "parlament.items" (a source may have several adapters)
|
||
schedule: ClassVar[str | None] # cron (UTC) or None = manual only
|
||
queue: ClassVar[str] = "scrape"
|
||
parser_version: ClassVar[int] = 1 # bump when parse() output changes → reparse
|
||
|
||
def start(self, ctx: RunContext) -> Iterable[Request]: ...
|
||
def parse(self, ctx: RunContext, response: Response) -> Iterable[Record | Request]: ...
|
||
```
|
||
|
||
* `start()` yields the entry requests (list endpoints, index pages, feeds). It may read
|
||
`ctx.state` (cursors) and `ctx.since`.
|
||
* `parse()` is **pure**: no DB access, no network. It yields records and follow-up requests
|
||
(detail pages). Dispatch on `response.request.kind` (`"list"`, `"detail"`, …).
|
||
* Anything that needs the database (e.g. "who sat in the Nationalrat on that day") is expressed
|
||
as resolution hints on refs and done by the Loader.
|
||
|
||
### 4.3 Request / Response
|
||
|
||
```python
|
||
@dataclass(frozen=True)
|
||
class Request:
|
||
url: str
|
||
kind: str = "default" # parse() dispatch key
|
||
method: str = "GET"
|
||
params: dict | None = None
|
||
json: Any = None
|
||
headers: dict | None = None
|
||
meta: dict = field(default_factory=dict) # passed through to parse (e.g. {"gp": "XXVIII"})
|
||
conditional: bool = True # send If-None-Match / If-Modified-Since
|
||
skip_unchanged: bool = True # don't reload records if body hash and parser_version unchanged
|
||
|
||
@dataclass
|
||
class Response:
|
||
request: Request
|
||
status: int
|
||
url: str
|
||
headers: Mapping[str, str]
|
||
body: bytes
|
||
fetched_at: datetime
|
||
sha256: str
|
||
fetch_id: int | None # row in the raw archive index
|
||
def text(self) -> str; def json(self) -> Any; def soup(self) -> BeautifulSoup
|
||
```
|
||
|
||
### 4.4 Fetcher (policy enforcement)
|
||
|
||
For every request the fetcher:
|
||
1. checks robots.txt (cached per origin) unless the URL starts with a `robots_exempt` prefix;
|
||
2. throttles per host (`max(spec.min_interval, robots crawl-delay)`);
|
||
3. sends an honest User-Agent with contact address, and conditional headers from the last fetch;
|
||
4. retries 429/5xx with exponential back-off, honouring `Retry-After`;
|
||
5. detects TDM reservations (`tdm-reservation` header, `/.well-known/tdmrep.json`) and records
|
||
them on the response (`response.tdm_reserved`) – the Loader then drops full text;
|
||
6. writes the body to the raw archive (`sha256`-addressed, gzip) **iff** `spec.keep_raw` and no
|
||
TDM reservation, and logs a `fetch_log` row in every case.
|
||
|
||
### 4.5 Records (typed, validated)
|
||
|
||
```python
|
||
Evidence(url, title=None, doc_type="web_page", published_at=None, excerpt=None, full_text=None,
|
||
quote=None, locator="", stance="supports", confidence=1.0, source=None)
|
||
PersonRecord(ref, full_name, given_name=None, family_name=None, …, aliases=(), social=())
|
||
OrgRecord(ref, name, short_name=None, kind="other", parent=None, color=None)
|
||
Claim(claim_type, summary_de, evidence, person=None, org=None, counterpart_person=None,
|
||
counterpart_org=None, occurred_at=None, valid_from=None, valid_to=None, summary_en=None,
|
||
data={}, topics=(), sensitive=False, review=False, key=None, facet=None)
|
||
# facets
|
||
Role(role, level, body=None, party=None, region=None, start=None, end=None)
|
||
VoteEvent(key, chamber, title, voted_on, result, stage=None, item_ref=None, roll_call=False, url=None, …)
|
||
VotePosition(event_key, position) # voter = claim.person or claim.org
|
||
Money(kind, amount, currency="EUR", payer=None, payee=None, payer_name=None, payee_name=None,
|
||
date=None, reference=None, category=None, period=None)
|
||
Event(key, kind, title, starts_at, ends_at=None, location=None, country=None, url=None, body=None)
|
||
Participation(event_key, role="participant")
|
||
Statement(context, url, said_at=None, medium=None, title=None, excerpt=None, platform=None)
|
||
Legal(stage, offense=None, authority=None, case_ref=None, stage_date=None)
|
||
Document(evidence) # a document without a claim (e.g. a headline);
|
||
# enrichers create claims from it later
|
||
```
|
||
|
||
Data minimisation is enforced by *schema*: e.g. `Money` has no address field, so a parser
|
||
cannot leak a donor's street address into the database even by accident.
|
||
|
||
### 4.6 Runner
|
||
|
||
```python
|
||
def run(adapter, mode="fetch", since=None):
|
||
ctx = RunContext(adapter, since, state=StateStore(adapter.name), stats=Counter())
|
||
queue = deque(adapter.start(ctx)); seen = set()
|
||
while queue:
|
||
req = queue.popleft()
|
||
if fingerprint(req) in seen: continue; seen.add(fingerprint(req))
|
||
resp = fetcher.fetch(req) if mode == "fetch" else archive.latest(req)
|
||
if resp is None: continue
|
||
if req.skip_unchanged and archive.already_parsed(resp, adapter.parser_version):
|
||
# records already loaded; still follow links so an interrupted run's children get fetched
|
||
queue.extend(r for r in adapter.parse(ctx, resp) if isinstance(r, Request)); continue
|
||
with db.transaction(): # one transaction per response
|
||
for item in adapter.parse(ctx, resp):
|
||
queue.append(item) if isinstance(item, Request) else loader.load(item, ctx)
|
||
archive.mark_parsed(resp, adapter.parser_version)
|
||
record run row with stats/errors
|
||
```
|
||
|
||
Per-response transactions mean a crash loses one page, never the whole run; errors in one
|
||
response are recorded and the run continues (error budget → run status `partial`).
|
||
|
||
### 4.7 Scheduling & deployment
|
||
|
||
* Procrastinate (Postgres-backed queue): the worker builds one periodic task per adapter from
|
||
`adapter.schedule`/`adapter.queue` – no hand-written task list.
|
||
* Queues `scrape`, `social`, `bulk`, `enrich` let heavy bulk jobs (procurement, ad archives) run
|
||
on their own worker (even another machine) without blocking light ones.
|
||
* One Docker image for api/worker/mcp/migrate; third-party adapter packages register via the
|
||
Python entry-point group `politrack.adapters`, so a separate deployment can add sources
|
||
without forking.
|
||
|
||
### 4.8 Enrichers
|
||
|
||
Same Loader, different input: an `Enricher` has `run(ctx) -> Iterable[Record]` over database
|
||
rows (documents not yet linked, claims not yet tagged…). Built-ins: mention linker,
|
||
event extractor (trips/meetings), legal-matter detector (→ review queue), topic tagger,
|
||
corroborator (merge near-duplicates, cluster news stories), say-vs-vote matcher, chunker +
|
||
embedder.
|
||
|
||
## 5. Serving
|
||
|
||
* **API**: FastAPI with Pydantic response models (OpenAPI is the contract). Standard filters
|
||
everywhere: `person`, `party`, `types`, `topics`, `since`, `until`, `exclude_sources`,
|
||
`min_trust`, `limit/offset`.
|
||
* **Web**: Next.js App Router + next-intl (de at `/`, en at `/en`); TypeScript types generated
|
||
from OpenAPI (`politrack openapi --ts`); preferences in a cookie so server components can
|
||
filter; a **renderer registry** maps claim types to components (adding a claim type = one
|
||
component).
|
||
* **Agents**: MCP tools mirroring the API; read-only SQL role; `COMMENT ON` everything;
|
||
`claim_with_evidence` view; `chunk` table for RAG with citations.
|
||
|
||
## 6. Testing strategy
|
||
|
||
| Level | What | How |
|
||
|---|---|---|
|
||
| Parser golden tests | each adapter's `parse()` | real responses saved as fixtures (`tests/fixtures/<adapter>/…`), expected records as JSON next to them; `UPDATE_GOLDEN=1` regenerates |
|
||
| Unit tests | text normalisation, resolvers, identity | plain pytest |
|
||
| Loader/integration | load records into an ephemeral Postgres, assert rows | `pytest -m integration` with `POLITRACK_TEST_DATABASE_URL` |
|
||
| API smoke | every endpoint against a seeded DB | FastAPI TestClient |
|
||
| Frontend | type check + build | `npm run typecheck && npm run build` |
|
||
|
||
## 7. Legal & ethical guard-rails in code
|
||
|
||
* Fetcher enforces robots/TDM/rate limits; `keep_raw`/`store_fulltext` per source.
|
||
* Record schemas exclude data we must not keep (addresses, private contact data).
|
||
* `sensitive=True` claims render the presumption-of-innocence notice unless stage is an
|
||
undisputed final conviction; auto-detected legal matters require `review`.
|
||
* Messengers: only *public broadcast channels* (e.g. public Telegram channels' web preview);
|
||
never private groups, never platforms whose terms forbid automated access (WhatsApp, X without
|
||
API, Facebook/Instagram).
|
||
|
||
## 8. Operations
|
||
|
||
* Services: `db` (Postgres 16 + pgvector), `migrate` (one-shot), `api`, `web`, `mcp`,
|
||
`worker-scrape`, `worker-bulk`, `worker-enrich`, optional `ollama`.
|
||
* Volumes: Postgres data, raw archive (`/data/raw`), Ollama models.
|
||
* Observability: `run` table, `politrack status`, `/api/v1/admin/runs` (token), structured logs.
|
||
* Backups: `pg_dump` + raw archive rsync (raw archive is optional to back up – it is a cache
|
||
that can be re-fetched, but it is also the audit trail).
|