Politrack: track Austrian politicians from public sources

Initial import of Politrack, a self-hosted service that tracks Austrian
politicians from federal to municipal level - roles, votes, money flows,
statements, meetings, social media and legal proceedings - with every claim
linked to its sources.

- backend: FastAPI + Postgres/pgvector; adapter framework (fetch -> parse ->
  load -> enrich) with robots.txt/TDM-aware fetching, a raw-response archive
  for reparsing, and a claim/evidence model with corroboration
- sources: Parliament, Wikidata, EU Parliament/Commission, Council of
  Ministers, Rechnungshof party reports and donations, procurement, RTR media
  transparency, political ad libraries, WKStA, news feeds, Wikipedia edits,
  Bluesky, Mastodon, public Telegram channels, YouTube and X (official API)
- web: Next.js frontend in German and English with typed API client
- mcp: MCP server for AI agents and RAG
- docs: architecture, data model, sources, legal notes, Unraid deployment and
  contributor guides; Docker Compose setup for Unraid

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
simonandClaude Opus 5.5 committed 2026-10-28 20:02:01 +01:00
commit 583ccba31c
187 files changed
+39890

No files matched your search

+68
View File
@@ -0,0 +1,68 @@
# Architecture (overview)
This page is the short version. The full design – quality goals, data model, the adapter framework down
to the request level, testing and guard-rails – is in
[architecture/01-design-from-scratch.md](architecture/01-design-from-scratch.md); the reasoning behind
each decision is in [architecture/02-comparison-and-decisions.md](architecture/02-comparison-and-decisions.md).
## Deployment
```
┌──────────────────────────── Unraid host ─────────────────────────────┐
Browser ──► reverse │ web (Next.js, de/en) ──► api (FastAPI, typed JSON) ──┐ │
(SWAG/NPM/Caddy) │ ▼ │
AI agents ─────────► │ mcp (MCP server, streamable HTTP) ──────────► PostgreSQL 16 │
BI / notebooks ────► │ (read-only role politrack_reader) ──────────► + pgvector, pg_trgm │
│ + procrastinate queue │
│ worker-scrape queues scrape,social (N replicas) ──┤ │
│ worker-bulk queue bulk (large downloads) ───────┤ │
│ worker-enrich queue enrich (linking, NLP, corroboration) ─┘ │
│ /data/raw raw-response archive (gzip, content-addressed) │
│ ollama (optional, bge-m3 embeddings) │
└────────────────────────────────────────────────────────────────────────┘
```
One backend image runs every role (API, MCP, workers, CLI). Workers can run on other machines that reach
the database (e.g. a scraper box with its own egress IP, a GPU box for embeddings).
## Pipeline in one picture
```
SourceSpec + Adapter.start() ─▶ Fetcher ─▶ raw archive ─▶ Adapter.parse() ─▶ Records ─▶ Loader ─▶ PostgreSQL
(code) robots.txt, throttle, (pure) typed resolve, ▲
TDM, conditional GET identity, │
evidence │
Enrichers (linker, events, legal, topics, corroborate) ───┘
```
* **Four strict stages** – fetch (policy + archive), parse (pure), load (single writer), enrich.
* **Everything is a claim with evidence**; the same fact from two sources is one claim (corroboration by
construction).
* **Sources are code** (`SourceSpec` next to the adapter), synced into the `source` table.
* **Re-parse without re-fetch** (`politrack reparse <adapter>`) after parser fixes.
* **Typed API** (Pydantic) → OpenAPI → generated TypeScript types for the web app.
## Why this stack
| Concern | Choice | Reason |
|---|---|---|
| Database | PostgreSQL 16 + pgvector + pg_trgm + unaccent | relational data, German full-text search, fuzzy names and vectors in one easily backed-up system that agents can query |
| Job queue | Procrastinate (Postgres-backed) | no Redis; cron-style periodic tasks, retries, queueing locks; horizontal workers |
| Backend | Python 3.12, httpx, protego, psycopg 3, FastAPI | best scraping/NLP ecosystem; one code base for collection and serving |
| Frontend | Next.js 15 + next-intl | server-side rendering, German at `/`, English at `/en` |
| Agents / RAG | MCP server, pgvector chunks, `COMMENT ON` everywhere, read-only SQL role | tools or plain SQL; every answer carries citation URLs |
| Embeddings | Ollama + bge-m3 (optional) | local, multilingual; full-text search works without it |
No Scrapy: its strengths (crawling at scale) are not our problem, while its model fights ours (pure
parsers, archive-first, per-publisher legal policy) – see ADR-20.
## Interfaces for other agents
1. **MCP** – `http://<host>:8001/mcp`: `search`, `find_person`, `get_person`, `person_network`,
`list_parties`, `get_party`, `find_organization`, `get_organization`, `list_claims`, `money_flows`,
`social_posts`, `wikipedia_edits`, `calendar`, `person_votes`, `former_politicians`, `list_topics`.
2. **REST** – `http://<host>:8000/api/docs` (OpenAPI 3).
3. **SQL** – role `politrack_reader`; start with the view `claim_with_evidence`; `\d+ claim` shows the
column comments.
4. **RAG** – table `chunk` (text, `tsv`, `embedding vector(1024)`), each chunk linked to a claim or
document for citations.