Politrack: track Austrian politicians from public sources
Initial import of Politrack, a self-hosted service that tracks Austrian politicians from federal to municipal level - roles, votes, money flows, statements, meetings, social media and legal proceedings - with every claim linked to its sources. - backend: FastAPI + Postgres/pgvector; adapter framework (fetch -> parse -> load -> enrich) with robots.txt/TDM-aware fetching, a raw-response archive for reparsing, and a claim/evidence model with corroboration - sources: Parliament, Wikidata, EU Parliament/Commission, Council of Ministers, Rechnungshof party reports and donations, procurement, RTR media transparency, political ad libraries, WKStA, news feeds, Wikipedia edits, Bluesky, Mastodon, public Telegram channels, YouTube and X (official API) - web: Next.js frontend in German and English with typed API client - mcp: MCP server for AI agents and RAG - docs: architecture, data model, sources, legal notes, Unraid deployment and contributor guides; Docker Compose setup for Unraid Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
commit
583ccba31c
187 files changed
+39890
No files matched your search
@@ -0,0 +1,68 @@
|
||||
# Architecture (overview)
|
||||
|
||||
This page is the short version. The full design – quality goals, data model, the adapter framework down
|
||||
to the request level, testing and guard-rails – is in
|
||||
[architecture/01-design-from-scratch.md](architecture/01-design-from-scratch.md); the reasoning behind
|
||||
each decision is in [architecture/02-comparison-and-decisions.md](architecture/02-comparison-and-decisions.md).
|
||||
|
||||
## Deployment
|
||||
|
||||
```
|
||||
┌──────────────────────────── Unraid host ─────────────────────────────┐
|
||||
Browser ──► reverse │ web (Next.js, de/en) ──► api (FastAPI, typed JSON) ──┐ │
|
||||
(SWAG/NPM/Caddy) │ ▼ │
|
||||
AI agents ─────────► │ mcp (MCP server, streamable HTTP) ──────────► PostgreSQL 16 │
|
||||
BI / notebooks ────► │ (read-only role politrack_reader) ──────────► + pgvector, pg_trgm │
|
||||
│ + procrastinate queue │
|
||||
│ worker-scrape queues scrape,social (N replicas) ──┤ │
|
||||
│ worker-bulk queue bulk (large downloads) ───────┤ │
|
||||
│ worker-enrich queue enrich (linking, NLP, corroboration) ─┘ │
|
||||
│ /data/raw raw-response archive (gzip, content-addressed) │
|
||||
│ ollama (optional, bge-m3 embeddings) │
|
||||
└────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
One backend image runs every role (API, MCP, workers, CLI). Workers can run on other machines that reach
|
||||
the database (e.g. a scraper box with its own egress IP, a GPU box for embeddings).
|
||||
|
||||
## Pipeline in one picture
|
||||
|
||||
```
|
||||
SourceSpec + Adapter.start() ─▶ Fetcher ─▶ raw archive ─▶ Adapter.parse() ─▶ Records ─▶ Loader ─▶ PostgreSQL
|
||||
(code) robots.txt, throttle, (pure) typed resolve, ▲
|
||||
TDM, conditional GET identity, │
|
||||
evidence │
|
||||
Enrichers (linker, events, legal, topics, corroborate) ───┘
|
||||
```
|
||||
|
||||
* **Four strict stages** – fetch (policy + archive), parse (pure), load (single writer), enrich.
|
||||
* **Everything is a claim with evidence**; the same fact from two sources is one claim (corroboration by
|
||||
construction).
|
||||
* **Sources are code** (`SourceSpec` next to the adapter), synced into the `source` table.
|
||||
* **Re-parse without re-fetch** (`politrack reparse <adapter>`) after parser fixes.
|
||||
* **Typed API** (Pydantic) → OpenAPI → generated TypeScript types for the web app.
|
||||
|
||||
## Why this stack
|
||||
|
||||
| Concern | Choice | Reason |
|
||||
|---|---|---|
|
||||
| Database | PostgreSQL 16 + pgvector + pg_trgm + unaccent | relational data, German full-text search, fuzzy names and vectors in one easily backed-up system that agents can query |
|
||||
| Job queue | Procrastinate (Postgres-backed) | no Redis; cron-style periodic tasks, retries, queueing locks; horizontal workers |
|
||||
| Backend | Python 3.12, httpx, protego, psycopg 3, FastAPI | best scraping/NLP ecosystem; one code base for collection and serving |
|
||||
| Frontend | Next.js 15 + next-intl | server-side rendering, German at `/`, English at `/en` |
|
||||
| Agents / RAG | MCP server, pgvector chunks, `COMMENT ON` everywhere, read-only SQL role | tools or plain SQL; every answer carries citation URLs |
|
||||
| Embeddings | Ollama + bge-m3 (optional) | local, multilingual; full-text search works without it |
|
||||
|
||||
No Scrapy: its strengths (crawling at scale) are not our problem, while its model fights ours (pure
|
||||
parsers, archive-first, per-publisher legal policy) – see ADR-20.
|
||||
|
||||
## Interfaces for other agents
|
||||
|
||||
1. **MCP** – `http://<host>:8001/mcp`: `search`, `find_person`, `get_person`, `person_network`,
|
||||
`list_parties`, `get_party`, `find_organization`, `get_organization`, `list_claims`, `money_flows`,
|
||||
`social_posts`, `wikipedia_edits`, `calendar`, `person_votes`, `former_politicians`, `list_topics`.
|
||||
2. **REST** – `http://<host>:8000/api/docs` (OpenAPI 3).
|
||||
3. **SQL** – role `politrack_reader`; start with the view `claim_with_evidence`; `\d+ claim` shows the
|
||||
column comments.
|
||||
4. **RAG** – table `chunk` (text, `tsv`, `embedding vector(1024)`), each chunk linked to a claim or
|
||||
document for citations.
|
||||
Reference in new issue
Block a user