Politrack: track Austrian politicians from public sources

Initial import of Politrack, a self-hosted service that tracks Austrian
politicians from federal to municipal level - roles, votes, money flows,
statements, meetings, social media and legal proceedings - with every claim
linked to its sources.

- backend: FastAPI + Postgres/pgvector; adapter framework (fetch -> parse ->
  load -> enrich) with robots.txt/TDM-aware fetching, a raw-response archive
  for reparsing, and a claim/evidence model with corroboration
- sources: Parliament, Wikidata, EU Parliament/Commission, Council of
  Ministers, Rechnungshof party reports and donations, procurement, RTR media
  transparency, political ad libraries, WKStA, news feeds, Wikipedia edits,
  Bluesky, Mastodon, public Telegram channels, YouTube and X (official API)
- web: Next.js frontend in German and English with typed API client
- mcp: MCP server for AI agents and RAG
- docs: architecture, data model, sources, legal notes, Unraid deployment and
  contributor guides; Docker Compose setup for Unraid

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/architecture.md

@@ -0,0 +1,68 @@
+# Architecture (overview)
+
+This page is the short version. The full design – quality goals, data model, the adapter framework down
+to the request level, testing and guard-rails – is in
+[architecture/01-design-from-scratch.md](architecture/01-design-from-scratch.md); the reasoning behind
+each decision is in [architecture/02-comparison-and-decisions.md](architecture/02-comparison-and-decisions.md).
+
+## Deployment
+
+```
+                       ┌──────────────────────────── Unraid host ─────────────────────────────┐
+  Browser ──► reverse  │  web (Next.js, de/en) ──► api (FastAPI, typed JSON) ──┐                │
+  (SWAG/NPM/Caddy)     │                                                       ▼                │
+  AI agents ─────────► │  mcp (MCP server, streamable HTTP) ──────────► PostgreSQL 16           │
+  BI / notebooks ────► │  (read-only role politrack_reader) ──────────►  + pgvector, pg_trgm    │
+                       │                                                  + procrastinate queue │
+                       │  worker-scrape  queues scrape,social (N replicas) ──┤                  │
+                       │  worker-bulk    queue bulk (large downloads) ───────┤                  │
+                       │  worker-enrich  queue enrich (linking, NLP, corroboration) ─┘          │
+                       │  /data/raw  raw-response archive (gzip, content-addressed)             │
+                       │  ollama (optional, bge-m3 embeddings)                                  │
+                       └────────────────────────────────────────────────────────────────────────┘
+```
+
+One backend image runs every role (API, MCP, workers, CLI). Workers can run on other machines that reach
+the database (e.g. a scraper box with its own egress IP, a GPU box for embeddings).
+
+## Pipeline in one picture
+
+```
+SourceSpec + Adapter.start() ─▶ Fetcher ─▶ raw archive ─▶ Adapter.parse() ─▶ Records ─▶ Loader ─▶ PostgreSQL
+            (code)               robots.txt, throttle,       (pure)          typed       resolve,      ▲
+                                 TDM, conditional GET                                    identity,     │
+                                                                                         evidence      │
+                                             Enrichers (linker, events, legal, topics, corroborate) ───┘
+```
+
+* **Four strict stages** – fetch (policy + archive), parse (pure), load (single writer), enrich.
+* **Everything is a claim with evidence**; the same fact from two sources is one claim (corroboration by
+  construction).
+* **Sources are code** (`SourceSpec` next to the adapter), synced into the `source` table.
+* **Re-parse without re-fetch** (`politrack reparse <adapter>`) after parser fixes.
+* **Typed API** (Pydantic) → OpenAPI → generated TypeScript types for the web app.
+
+## Why this stack
+
+| Concern | Choice | Reason |
+|---|---|---|
+| Database | PostgreSQL 16 + pgvector + pg_trgm + unaccent | relational data, German full-text search, fuzzy names and vectors in one easily backed-up system that agents can query |
+| Job queue | Procrastinate (Postgres-backed) | no Redis; cron-style periodic tasks, retries, queueing locks; horizontal workers |
+| Backend | Python 3.12, httpx, protego, psycopg 3, FastAPI | best scraping/NLP ecosystem; one code base for collection and serving |
+| Frontend | Next.js 15 + next-intl | server-side rendering, German at `/`, English at `/en` |
+| Agents / RAG | MCP server, pgvector chunks, `COMMENT ON` everywhere, read-only SQL role | tools or plain SQL; every answer carries citation URLs |
+| Embeddings | Ollama + bge-m3 (optional) | local, multilingual; full-text search works without it |
+
+No Scrapy: its strengths (crawling at scale) are not our problem, while its model fights ours (pure
+parsers, archive-first, per-publisher legal policy) – see ADR-20.
+
+## Interfaces for other agents
+
+1. **MCP** – `http://<host>:8001/mcp`: `search`, `find_person`, `get_person`, `person_network`,
+   `list_parties`, `get_party`, `find_organization`, `get_organization`, `list_claims`, `money_flows`,
+   `social_posts`, `wikipedia_edits`, `calendar`, `person_votes`, `former_politicians`, `list_topics`.
+2. **REST** – `http://<host>:8000/api/docs` (OpenAPI 3).
+3. **SQL** – role `politrack_reader`; start with the view `claim_with_evidence`; `\d+ claim` shows the
+   column comments.
+4. **RAG** – table `chunk` (text, `tsv`, `embedding vector(1024)`), each chunk linked to a claim or
+   document for citations.

docs/architecture/01-design-from-scratch.md

@@ -0,0 +1,325 @@
+# Politrack – clean-room design
+
+> Written as if no code existed. Inputs were only the requirements: track what Austrian
+> politicians (federal → municipal) do, from many heterogeneous public sources, legally, with
+> every fact backed by links; user-selectable trust & topics; person/party overviews; calendar
+> and timeline; multilingual (de default, en); agent/RAG access; runs on an Unraid homelab in
+> Docker; scrapers deployable separately; easy for contributors to extend.
+>
+> [02-comparison-and-decisions.md](02-comparison-and-decisions.md) compares this design with
+> the first implementation and records which decisions won.
+
+## 1. Quality goals (in priority order)
+
+1. **Trustworthiness** – no fact without provenance; never present something as more certain
+   than its sources; sensitive facts (criminal matters) are gated. Precision beats recall.
+2. **Legal compliance by construction** – the framework, not the individual scraper author,
+   enforces robots.txt, TDM reservations, rate limits, storage limits and data minimisation.
+3. **Reproducibility** – any derived row can be explained ("which source, which response,
+   which parser version") and regenerated from archived raw data after a parser fix, without
+   hitting the source again.
+4. **Extensibility** – adding a source must touch exactly one module (+ a fixture-based test
+   and a docs line). No edits to SQL seeds, schedulers or registries elsewhere.
+5. **Operability on a homelab** – few moving parts (one database, one image), self-healing
+   schedules, everything inspectable with SQL.
+
+## 2. System context
+
+```
+               ┌───────────── public sources (APIs, open data, feeds, sitemaps, archives) ─────────────┐
+               ▼                                                                                        │
+      ┌────────────────┐  Request   ┌──────────┐ Response  ┌──────────┐ Records ┌────────┐            │
+      │ Adapter.start()│──────────▶ │ Fetcher  │─────────▶ │ Adapter. │───────▶ │ Loader │──▶ Postgres│
+      │ (per source)   │◀────────── │ (policy, │  archive  │ parse()  │ FollowUp│        │            │
+      └────────────────┘  FollowUp  │ archive) │──▶ Raw    └──────────┘         └────────┘            │
+                                    └──────────┘   Archive (blobs + fetch_log)      ▲                 │
+                                                                                      │ Records         │
+                                      Enrichers (linking, events, legal, topics, ─────┘                 │
+                                      corroboration, embeddings) read Postgres                          │
+                                                                                                        │
+      Serving: FastAPI (typed JSON, OpenAPI) ─▶ Next.js web (de/en)      MCP server ─▶ AI agents       │
+               read-only SQL role ─▶ notebooks/BI/agents                 pgvector chunks ─▶ RAG         │
+```
+
+Four strictly separated stages:
+
+| Stage | Input → output | Side effects | Determinism |
+|---|---|---|---|
+| **Fetch** | `Request` → `Response` | network; raw archive write | no |
+| **Parse** | `Response` → `Record`s and follow-up `Request`s | none (pure) | yes, versioned |
+| **Load** | `Record` → canonical rows | DB writes (single code path) | yes, idempotent |
+| **Enrich** | canonical rows → `Record`s | via Loader only | yes, idempotent |
+
+Because parse is pure and responses are archived, `politrack reparse <adapter>` rebuilds data
+after a parser fix without re-fetching. Because load is the only writer, invariants (no claim
+without evidence, data minimisation, identity rules) live in one place.
+
+## 3. Data model
+
+### 3.1 Principles
+
+* **Claim/evidence**: every user-visible fact is a `claim` with ≥1 `claim_evidence` row pointing
+  to a `document` (a URL) published by a `source`. Claims from different sources about the same
+  fact collapse into one claim with several evidence rows ("corroborated").
+* **Typed facets**: claims of some types carry a structured facet row (role, vote position,
+  money flow, event participation, statement, legal proceeding). Facets make analytics and
+  correlations possible; the claim makes provenance and display uniform.
+* **Identifiers instead of ID columns**: entities get any number of external identifiers in one
+  table (`scheme`, `value`), e.g. `parlament:2339`, `wikidata:Q123`, `ep:197479`,
+  `bluesky:did:plc:…`, `telegram:HKickl`, `uid:ATU12345678`, `rtr:7846`. New sources add schemes,
+  not columns.
+* **Bitemporal enough**: claims have `occurred_at` (point events) or `valid_from/valid_to`
+  (states such as mandates); documents have `published_at`; everything has `created_at/updated_at`.
+
+### 3.2 Tables (abridged; full DDL in `backend/migrations`)
+
+```
+source(slug PK-ish, name, kind, homepage, license, access, trust, store_fulltext, keep_raw, tos_notes, …)
+fetch(id, adapter, source_id, request_fp, url, method, status, sha256, size, content_type,
+      etag, last_modified, fetched_at, request jsonb, parsed_sha256, parser_version)      -- raw archive index
+document(id, source_id, url UNIQUE, title, doc_type, lang, published_at, excerpt, full_text,
+         fetch_id, story_id, meta jsonb)
+person(id, slug, full_name, given_name, family_name, prefixes…, birth_date, death_date, gender, photo_url)
+organization(id, slug, name, short_name, kind, parent_id, color, …)
+identifier(entity_type, entity_id, scheme, value, source_slug, verified)  UNIQUE(entity_type, scheme, value)
+alias(entity_type, entity_id, alias, alias_norm)
+social_account(id, person_id|organization_id, platform, handle, url, verified, source_slug)
+claim(id, claim_type, fingerprint UNIQUE, person_id, organization_id, counterpart_person_id,
+      counterpart_org_id, occurred_at, valid_from, valid_to, summary_de, summary_en, data jsonb,
+      status, sensitive, review_status, merged_into, created_by, …)
+claim_evidence(claim_id, document_id, stance, quote, locator, extracted_by, confidence)
+role / vote_event / vote_position / money_flow / event / event_participant / statement /
+legal_proceeding                                                             -- facets
+topic, topic_assignment, story, chunk(text, tsv, embedding vector(1024))
+run(id, adapter, started_at, finished_at, status, stats, error), state(adapter, key, value)
+```
+
+### 3.3 Claim identity
+
+The *Loader* (not the scraper) computes a claim's fingerprint from **resolved entity ids** and
+the semantically identifying fields of its type:
+
+| claim type | identity |
+|---|---|
+| role | person, body, canonical role, start date |
+| vote | vote event key, voter (person or party) |
+| donation / money_flow | kind, payer, payee, amount, date, external reference |
+| statement | person, URL |
+| media_mention | URL, subject |
+| meeting / travel | person, anchor (counterpart surname or country), ISO week |
+| legal_proceeding, others | explicit key from the adapter |
+
+Using resolved ids means a Wikidata mandate and a Parliament mandate of the same person, body
+and start date *are the same claim* by construction. A fuzzy **corroboration** pass handles
+near-duplicates (dates off by days, wording differences) by merging claims (`merged_into`).
+
+### 3.4 Entity resolution
+
+Records refer to entities through `PersonRef` / `OrgRef`:
+
+```python
+PersonRef(ids={"parlament": "2339"}, name="Selma Yildirim")             # strong id
+PersonRef(name="Auer", hints={"surname_in_body": "nationalrat", "on": date(2025, 9, 24)})
+OrgRef(name="Hypo Tirol Bank AG", kind="company", ids={"uid": "ATU…"})
+```
+
+Resolution order: (1) any strong identifier → entity; (2) strategy named in hints
+(e.g. *surname among members of body on date*); (3) exact normalised alias; (4) for
+organisations: normalised legal name (legal-form suffixes stripped); (5) create — persons are
+only created by "authoritative" adapters (`creates_persons=True`), other adapters leave
+unresolvable person refs unlinked and count them. Ambiguity never guesses.
+
+## 4. The adapter framework (the nitty-gritty)
+
+### 4.1 SourceSpec – single source of truth for a publisher
+
+```python
+@dataclass(frozen=True)
+class SourceSpec:
+    slug: str                 # "parlament"
+    name: str
+    kind: Literal["official", "ngo", "media", "social", "aggregator", "user"]
+    homepage: str
+    license: str
+    access: Literal["api", "bulk", "rss", "scrape", "manual"]
+    trust: int                # default 0–100, editorial
+    store_fulltext: bool      # may we keep article/body text?
+    keep_raw: bool = True     # may we archive raw responses? (False for TDM-reserved)
+    min_interval: float = 2.0 # seconds between requests to one host
+    robots_exempt: tuple[str, ...] = ()   # URL prefixes of documented APIs; justify in tos_notes
+    tos_notes: str = ""
+```
+
+Specs live next to their adapter. `politrack sources sync` (run by `migrate`) upserts them into
+the `source` table, so SQL never needs editing to add a source.
+
+### 4.2 Adapter
+
+```python
+class Adapter:
+    spec: ClassVar[SourceSpec]
+    name: ClassVar[str]                 # "parlament.items" (a source may have several adapters)
+    schedule: ClassVar[str | None]      # cron (UTC) or None = manual only
+    queue: ClassVar[str] = "scrape"
+    parser_version: ClassVar[int] = 1   # bump when parse() output changes → reparse
+
+    def start(self, ctx: RunContext) -> Iterable[Request]: ...
+    def parse(self, ctx: RunContext, response: Response) -> Iterable[Record | Request]: ...
+```
+
+* `start()` yields the entry requests (list endpoints, index pages, feeds). It may read
+  `ctx.state` (cursors) and `ctx.since`.
+* `parse()` is **pure**: no DB access, no network. It yields records and follow-up requests
+  (detail pages). Dispatch on `response.request.kind` (`"list"`, `"detail"`, …).
+* Anything that needs the database (e.g. "who sat in the Nationalrat on that day") is expressed
+  as resolution hints on refs and done by the Loader.
+
+### 4.3 Request / Response
+
+```python
+@dataclass(frozen=True)
+class Request:
+    url: str
+    kind: str = "default"           # parse() dispatch key
+    method: str = "GET"
+    params: dict | None = None
+    json: Any = None
+    headers: dict | None = None
+    meta: dict = field(default_factory=dict)   # passed through to parse (e.g. {"gp": "XXVIII"})
+    conditional: bool = True        # send If-None-Match / If-Modified-Since
+    skip_unchanged: bool = True     # don't reload records if body hash and parser_version unchanged
+
+@dataclass
+class Response:
+    request: Request
+    status: int
+    url: str
+    headers: Mapping[str, str]
+    body: bytes
+    fetched_at: datetime
+    sha256: str
+    fetch_id: int | None            # row in the raw archive index
+    def text(self) -> str; def json(self) -> Any; def soup(self) -> BeautifulSoup
+```
+
+### 4.4 Fetcher (policy enforcement)
+
+For every request the fetcher:
+1. checks robots.txt (cached per origin) unless the URL starts with a `robots_exempt` prefix;
+2. throttles per host (`max(spec.min_interval, robots crawl-delay)`);
+3. sends an honest User-Agent with contact address, and conditional headers from the last fetch;
+4. retries 429/5xx with exponential back-off, honouring `Retry-After`;
+5. detects TDM reservations (`tdm-reservation` header, `/.well-known/tdmrep.json`) and records
+   them on the response (`response.tdm_reserved`) – the Loader then drops full text;
+6. writes the body to the raw archive (`sha256`-addressed, gzip) **iff** `spec.keep_raw` and no
+   TDM reservation, and logs a `fetch_log` row in every case.
+
+### 4.5 Records (typed, validated)
+
+```python
+Evidence(url, title=None, doc_type="web_page", published_at=None, excerpt=None, full_text=None,
+         quote=None, locator="", stance="supports", confidence=1.0, source=None)
+PersonRecord(ref, full_name, given_name=None, family_name=None, …, aliases=(), social=())
+OrgRecord(ref, name, short_name=None, kind="other", parent=None, color=None)
+Claim(claim_type, summary_de, evidence, person=None, org=None, counterpart_person=None,
+      counterpart_org=None, occurred_at=None, valid_from=None, valid_to=None, summary_en=None,
+      data={}, topics=(), sensitive=False, review=False, key=None, facet=None)
+# facets
+Role(role, level, body=None, party=None, region=None, start=None, end=None)
+VoteEvent(key, chamber, title, voted_on, result, stage=None, item_ref=None, roll_call=False, url=None, …)
+VotePosition(event_key, position)            # voter = claim.person or claim.org
+Money(kind, amount, currency="EUR", payer=None, payee=None, payer_name=None, payee_name=None,
+      date=None, reference=None, category=None, period=None)
+Event(key, kind, title, starts_at, ends_at=None, location=None, country=None, url=None, body=None)
+Participation(event_key, role="participant")
+Statement(context, url, said_at=None, medium=None, title=None, excerpt=None, platform=None)
+Legal(stage, offense=None, authority=None, case_ref=None, stage_date=None)
+Document(evidence)                           # a document without a claim (e.g. a headline);
+                                             # enrichers create claims from it later
+```
+
+Data minimisation is enforced by *schema*: e.g. `Money` has no address field, so a parser
+cannot leak a donor's street address into the database even by accident.
+
+### 4.6 Runner
+
+```python
+def run(adapter, mode="fetch", since=None):
+    ctx = RunContext(adapter, since, state=StateStore(adapter.name), stats=Counter())
+    queue = deque(adapter.start(ctx)); seen = set()
+    while queue:
+        req = queue.popleft()
+        if fingerprint(req) in seen: continue; seen.add(fingerprint(req))
+        resp = fetcher.fetch(req) if mode == "fetch" else archive.latest(req)
+        if resp is None: continue
+        if req.skip_unchanged and archive.already_parsed(resp, adapter.parser_version):
+            # records already loaded; still follow links so an interrupted run's children get fetched
+            queue.extend(r for r in adapter.parse(ctx, resp) if isinstance(r, Request)); continue
+        with db.transaction():                 # one transaction per response
+            for item in adapter.parse(ctx, resp):
+                queue.append(item) if isinstance(item, Request) else loader.load(item, ctx)
+            archive.mark_parsed(resp, adapter.parser_version)
+    record run row with stats/errors
+```
+
+Per-response transactions mean a crash loses one page, never the whole run; errors in one
+response are recorded and the run continues (error budget → run status `partial`).
+
+### 4.7 Scheduling & deployment
+
+* Procrastinate (Postgres-backed queue): the worker builds one periodic task per adapter from
+  `adapter.schedule`/`adapter.queue` – no hand-written task list.
+* Queues `scrape`, `social`, `bulk`, `enrich` let heavy bulk jobs (procurement, ad archives) run
+  on their own worker (even another machine) without blocking light ones.
+* One Docker image for api/worker/mcp/migrate; third-party adapter packages register via the
+  Python entry-point group `politrack.adapters`, so a separate deployment can add sources
+  without forking.
+
+### 4.8 Enrichers
+
+Same Loader, different input: an `Enricher` has `run(ctx) -> Iterable[Record]` over database
+rows (documents not yet linked, claims not yet tagged…). Built-ins: mention linker,
+event extractor (trips/meetings), legal-matter detector (→ review queue), topic tagger,
+corroborator (merge near-duplicates, cluster news stories), say-vs-vote matcher, chunker +
+embedder.
+
+## 5. Serving
+
+* **API**: FastAPI with Pydantic response models (OpenAPI is the contract). Standard filters
+  everywhere: `person`, `party`, `types`, `topics`, `since`, `until`, `exclude_sources`,
+  `min_trust`, `limit/offset`.
+* **Web**: Next.js App Router + next-intl (de at `/`, en at `/en`); TypeScript types generated
+  from OpenAPI (`politrack openapi --ts`); preferences in a cookie so server components can
+  filter; a **renderer registry** maps claim types to components (adding a claim type = one
+  component).
+* **Agents**: MCP tools mirroring the API; read-only SQL role; `COMMENT ON` everything;
+  `claim_with_evidence` view; `chunk` table for RAG with citations.
+
+## 6. Testing strategy
+
+| Level | What | How |
+|---|---|---|
+| Parser golden tests | each adapter's `parse()` | real responses saved as fixtures (`tests/fixtures/<adapter>/…`), expected records as JSON next to them; `UPDATE_GOLDEN=1` regenerates |
+| Unit tests | text normalisation, resolvers, identity | plain pytest |
+| Loader/integration | load records into an ephemeral Postgres, assert rows | `pytest -m integration` with `POLITRACK_TEST_DATABASE_URL` |
+| API smoke | every endpoint against a seeded DB | FastAPI TestClient |
+| Frontend | type check + build | `npm run typecheck && npm run build` |
+
+## 7. Legal & ethical guard-rails in code
+
+* Fetcher enforces robots/TDM/rate limits; `keep_raw`/`store_fulltext` per source.
+* Record schemas exclude data we must not keep (addresses, private contact data).
+* `sensitive=True` claims render the presumption-of-innocence notice unless stage is an
+  undisputed final conviction; auto-detected legal matters require `review`.
+* Messengers: only *public broadcast channels* (e.g. public Telegram channels' web preview);
+  never private groups, never platforms whose terms forbid automated access (WhatsApp, X without
+  API, Facebook/Instagram).
+
+## 8. Operations
+
+* Services: `db` (Postgres 16 + pgvector), `migrate` (one-shot), `api`, `web`, `mcp`,
+  `worker-scrape`, `worker-bulk`, `worker-enrich`, optional `ollama`.
+* Volumes: Postgres data, raw archive (`/data/raw`), Ollama models.
+* Observability: `run` table, `politrack status`, `/api/v1/admin/runs` (token), structured logs.
+* Backups: `pg_dump` + raw archive rsync (raw archive is optional to back up – it is a cache
+  that can be re-fetched, but it is also the audit trail).

docs/architecture/02-comparison-and-decisions.md

@@ -0,0 +1,92 @@
+# Clean-room design vs. first implementation – decisions
+
+The first implementation (v1) grew iteratively. The clean-room design
+([01-design-from-scratch.md](01-design-from-scratch.md)) was written without looking at it.
+This document compares them area by area and records the decision taken for v2 (the current
+code). Format: lightweight ADRs.
+
+## Summary
+
+| # | Area | v1 (first implementation) | Clean-room design | **Decision (v2)** |
+|---|---|---|---|---|
+| 1 | Pipeline shape | each scraper fetches, parses and writes in one method | fetch → archive → pure parse → loader | **Clean-room**. Bugs found during v1 testing (vote parser, roll calls, offence regex) required resetting state and re-fetching; with an archive they are a `reparse`. |
+| 2 | Writing to the DB | scrapers call `store.record_claim(...)` with hand-made fingerprint keys | typed records; Loader computes identity from resolved entity ids | **Clean-room**, keeping v1's battle-tested upsert SQL inside the Loader. |
+| 3 | Source definition | SQL seed + scraper class + worker schedule + YAML, four places | `SourceSpec` in code, synced to DB; schedule on the adapter | **Clean-room**. One module per source. |
+| 4 | External ids | columns `parlament_pad`, `wikidata_qid` | `identifier(scheme, value)` table | **Clean-room**. Needed anyway for EP, Bluesky DIDs, Telegram, company UIDs, RTR ids. |
+| 5 | Entity references | scrapers look persons up themselves (SQL in scrapers) | `PersonRef`/`OrgRef` with ids/name/hints, resolved by the Loader | **Clean-room**. Makes parsers pure and testable. |
+| 6 | Change detection | per-scraper ad-hoc signatures in `scrape_state` | generic: body hash + parser version per request fingerprint; conditional GET | **Clean-room**; v1's cursor state store kept for real cursors (dates, ids). |
+| 7 | Policy enforcement | `PoliteClient` (robots, TDM, throttle, retries) + per-call `documented_api=True` | same, but exemptions declared on `SourceSpec.robots_exempt` | **Merge**: v1 client code + declarative exemptions. |
+| 8 | Corroboration | post-hoc merge engine (roles, stories, events) | identity by construction + fuzzy merge pass | **Both**: identity by construction removes most duplicates; v1's merge engine handles near-duplicates. |
+| 9 | Activity status | `person_status` view with verified flag and term windows | not specified in detail | **v1** (proven on real data: Van der Bellen/Mikl-Leitner cases). |
+| 10 | Legal matters | review queue, innocence-notice rule, official immunity data | same principles | **v1** implementation, moved behind the Loader (`review=True`). |
+| 11 | Scheduling | Procrastinate, tasks written by hand | Procrastinate, tasks generated from adapters; extra `bulk`/`social` queues | **Clean-room**. |
+| 12 | Enrichment | functions writing claims directly | enrichers emitting records to the Loader | **Clean-room**. |
+| 13 | API contract | handwritten dict responses; hand-maintained TS types | Pydantic response models, generated TS types | **Clean-room**, generated with a small built-in generator (`politrack openapi --ts`) so it works without npm. |
+| 14 | Frontend claim rendering | one generic `ClaimList` with `if` branches | renderer registry per claim type | **Clean-room** (generic fallback + specific renderers). |
+| 15 | Tests | inline-string unit tests | golden tests from real fixtures + unit + integration + API smoke | **Clean-room**, keeping v1's unit tests. |
+| 16 | Data minimisation | sanitiser function in the donations scraper | enforced by record schemas | **Clean-room** + keep the sanitiser (defence in depth). |
+| 17 | Storage of raw data | none | content-addressed raw archive, per-source `keep_raw` | **Clean-room**. |
+| 18 | Plugins | none | entry points `politrack.adapters` | **Clean-room** (cheap to add, enables separate deployments). |
+| 19 | Stack choices (Postgres+pgvector, FastAPI, Next.js+next-intl, MCP, Docker Compose) | ✓ | ✓ | **Same in both** – confirmed. |
+| 20 | Scrapy vs. own framework | own | own (Scrapy considered) | **Own**. See ADR-20. |
+
+## Selected ADRs
+
+### ADR-1: Archive raw responses and make parsers pure
+**Context.** In v1 three parser bugs were found after a full run; fixing them required
+deleting state, rows and refetching ~1,000 pages (and politely waiting).
+**Decision.** Every fetch is logged; bodies are archived (gzip, sha256-addressed) unless the
+source forbids it (`keep_raw=False`, TDM reservation). Parsers are pure functions of the
+response. `politrack reparse <adapter>` replays the archive.
+**Consequences.** Disk usage (Parliament JSON + protocols ≈ a few GB/year; bulk sources more –
+retention configurable). Parsers become trivially unit-testable with real fixtures.
+
+### ADR-2: The Loader is the only writer
+**Decision.** Adapters and enrichers yield typed records; `Loader.load()` resolves references,
+computes identity, upserts documents/claims/evidence/facets, assigns topics and refreshes
+status. Invariants (no claim without evidence, full text only where licensed, review gating)
+are enforced there.
+
+### ADR-4: `identifier` table
+**Decision.** `identifier(entity_type, entity_id, scheme, value)`; schemes are free-form
+strings documented in `docs/data-model.md` (`parlament`, `wikidata`, `ep`, `bluesky`,
+`mastodon`, `telegram`, `youtube`, `uid`, `firmenbuch`, `rtr`, `eu_transparency`, …).
+
+### ADR-6: Generic change detection
+**Decision.** The fetcher stores ETag/Last-Modified per request fingerprint and sends
+conditional requests; the runner skips `parse()` when the body hash and parser version equal the
+last successful parse. Adapters no longer keep "seen" dictionaries.
+
+### ADR-13: Typed API, generated TS types without npm
+**Context.** The development network blocks npm; contributors should still be able to
+regenerate types. **Decision.** `politrack openapi` writes `openapi.json` and
+`web/src/lib/api-types.ts` from the FastAPI app using a small generator for the subset of
+JSON-Schema FastAPI emits.
+
+### ADR-20: No Scrapy
+**Context.** Scrapy offers robots middleware, throttling, HTTP cache and pipelines.
+**Decision.** Keep a small own framework (~600 lines): our sources are mostly APIs and bulk
+files, we need DB-integrated change detection, Procrastinate scheduling, TDM handling and pure
+parsers with typed records. Scrapy's Twisted runtime and project layout would add more
+complexity than it removes. Revisit if we ever crawl large websites.
+
+### ADR-21: Messaging platforms
+**Decision.** Public Telegram *channels* via their public web preview (`t.me/s/<channel>`,
+no robots restrictions, read-only, low frequency) and only for handles from verifiable
+sources (Wikidata P3789, official websites). WhatsApp Channels and Signal are not collected:
+WhatsApp offers no public read access and its terms prohibit automated collection; Signal has
+no public channels and groups are private by design.
+
+### ADR-22: X only through the official API, opt-in
+**Context.** Many Austrian politicians are most active on X; proxies such as Nitter work by violating
+X's terms. **Decision.** `social.x` uses the official API v2 with the operator's bearer token and does
+nothing without one. It keeps no raw archive and no full text (`keep_raw=False`, `store_fulltext=False`).
+The Developer Agreement requires removing posts deleted on X; until a deletion sync exists the adapter is
+documented as not production-ready.
+
+### ADR-23: Organisation-owned accounts
+**Context.** Party and institution channels (MFG on Telegram, NEOS and the Federal President on Bluesky)
+belong to organisations, not people. **Decision.** `social_account` and statements may belong to an
+organisation (`statement.person_id` NULL, `claim.organization_id` set); the social feed shows them, and
+party pages include the party's own channels. Curated accounts live in `config/social_accounts.yaml`
+(imported by `config.social_accounts`), each with the official page that links it.

docs/contributing/adapters.md

@@ -0,0 +1,291 @@
+# Writing a source adapter
+
+An **adapter** turns one publisher's data into typed records. It never touches HTTP or the database
+directly: the framework fetches (politely, legally, archived) and the Loader writes. Your job is two
+functions, `start()` and `parse()`, and a description of the publisher (`SourceSpec`).
+
+```
+start(ctx)          -> Request ...                 entry points (list pages, feeds, API queries)
+      │  Fetcher: robots.txt, throttle, TDM check, conditional GET, retries, raw archive
+      ▼
+parse(ctx, resp)    -> Record | Request ...        PURE: no DB, no network, no clock-dependent output
+      │  Loader: entity resolution, claim identity, evidence, facets, corroboration status
+      ▼
+PostgreSQL
+```
+
+Read [architecture/01-design-from-scratch.md](../architecture/01-design-from-scratch.md) §4 for the
+reasoning; this page is the how-to.
+
+## 0. Before writing code: the legal checklist
+
+Answer these in the pull request (they end up in `SourceSpec.tos_notes`):
+
+1. **Is there an official API, bulk download or feed?** Use it instead of scraping HTML.
+2. **robots.txt** – the Fetcher enforces it. Never work around it. If a documented public API is
+   listed in robots.txt only because of generic crawler rules (Wikidata SPARQL, the MediaWiki API), add
+   the API prefix to `robots_exempt` and cite the operator's API policy in `tos_notes`.
+3. **Terms of service / licence** – may automated access happen? May we keep full text? If not,
+   `store_fulltext=False` (only title, short excerpt and link are kept) and usually `keep_raw=False`.
+4. **TDM reservation** (§42h UrhG, W3C TDMRep) – detected automatically per origin. If set, no text and no
+   raw archive are kept, whatever the spec says.
+5. **Personal data** – only data about people in their public role. There is no address field in any
+   record type; don't smuggle one into `data`.
+6. **No circumvention** – no paywall tricks, no mirrors/proxies of platforms whose ToS forbid scraping
+   (e.g. Nitter for X), no logged-in sessions, no private groups.
+
+If any answer is "no" or "unclear", document the source in [docs/sources.md](../sources.md) under
+"Approaches we deliberately do not use" instead.
+
+## 1. Where things go
+
+```
+backend/politrack/sources/
+  my_source.py              one module per publisher (or a package for large ones, see parlament/)
+backend/tests/fixtures/<adapter name>/
+  <case>.request.json       captured request (url, kind, meta)
+  <case>.html|json|xml|bin  captured response body
+  <case>.expected.json      golden output of parse()
+```
+
+Modules under `politrack/sources` and `politrack/enrich` are imported automatically; `@register` is all
+it takes. External packages can add adapters through the `politrack.adapters` entry-point group (see
+`backend/pyproject.toml`).
+
+## 2. A complete example
+
+A (fictitious) state parliament publishes its members as JSON and a page per member:
+
+```python
+"""Landtag Musterland: members (JSON API) and their declared side jobs (HTML)."""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+from typing import Any
+
+from politrack.core import Adapter, Request, Response, RunContext, SourceSpec, register
+from politrack.core.records import Claim, Evidence, OrgRef, PersonRecord, PersonRef, Role
+from politrack.text import excerpt
+
+BASE = "https://www.landtag-musterland.at"
+SPEC = SourceSpec(
+    slug="landtag-musterland",              # stable; referenced by Evidence.source and user filters
+    name="Landtag Musterland",
+    kind="official",                        # official | ngo | media | social | aggregator | user | derived
+    homepage=BASE,
+    license="Amtliches Werk (§7 UrhG)",
+    access="api",                           # api | bulk | rss | scrape | manual | derived
+    trust=90,                               # default trust 0-100 (users can override)
+    min_interval=2.0,                       # seconds between requests to this host
+    tos_notes="Open JSON API, no terms restricting reuse; robots.txt allows /api/.",
+)
+
+
+@register
+class MusterlandMembers(Adapter):
+    spec = SPEC
+    name = "musterland.members"             # <source>.<dataset>; also the CLI/worker task name
+    description = "Members of the Landtag Musterland with mandates and side jobs"
+    schedule = "15 4 * * *"                 # cron (UTC); None = manual only
+    queue = "scrape"                        # scrape | social | bulk
+    creates_persons = True                  # authoritative member list: may create persons
+    parser_version = 1                      # bump when parse() output changes, then `politrack reparse`
+
+    def start(self, ctx: RunContext) -> Iterator[Request]:
+        yield Request(url=f"{BASE}/api/members", params={"period": ctx.params.get("period", "current")},
+                      kind="list")
+
+    def parse(self, ctx: RunContext, response: Response) -> Iterator[Any]:
+        if response.request.kind == "list":
+            for m in response.json()["members"]:
+                ref = PersonRef(ids={"landtag-musterland": str(m["id"])}, name=m["name"])
+                yield PersonRecord(ref=ref, full_name=m["name"], photo_url=m.get("photo"))
+                yield Claim(
+                    claim_type="role", person=ref,
+                    summary_de=f"{m['name']} ist Abgeordnete:r zum Landtag Musterland ({m['party']})",
+                    summary_en=f"{m['name']} is a member of the Musterland state parliament ({m['party']})",
+                    valid_from=m.get("since"),
+                    evidence=[Evidence(url=f"{BASE}/abgeordnete/{m['id']}", doc_type="api_record",
+                                       title=m["name"])],
+                    facet=Role(role="mp", level="state", region="Musterland",
+                               body=OrgRef(slug="landtag-musterland", name="Landtag Musterland", kind="parliament"),
+                               party=OrgRef(name=m["party"], kind="party", create=False),
+                               start=m.get("since")),
+                )
+                # follow-up request; meta travels with it to parse()
+                yield Request(url=f"{BASE}/abgeordnete/{m['id']}", kind="profile",
+                              meta={"member_id": str(m["id"]), "name": m["name"]})
+            return
+
+        # kind == "profile": HTML page with a side-job table
+        for row in response.soup.select("table.nebentaetigkeiten tr")[1:]:
+            employer, function = (td.get_text(" ", strip=True) for td in row.find_all("td")[:2])
+            yield Claim(
+                claim_type="side_job",
+                person=PersonRef(ids={"landtag-musterland": response.meta["member_id"]}),
+                counterpart_org=OrgRef(name=employer, kind="company"),
+                key=("musterland", response.meta["member_id"], employer, function),   # no facet -> explicit key
+                summary_de=f"{response.meta['name']}: {function} bei {employer}",
+                evidence=[Evidence(url=response.url, doc_type="register_entry", quote=excerpt(row.get_text(" "), 300))],
+            )
+```
+
+Run it:
+
+```bash
+docker compose exec api politrack sources sync                       # registers SPEC in the DB
+docker compose exec api politrack run musterland.members --max-requests 20 -p period=current
+docker compose exec api politrack status                             # run stats, errors
+```
+
+The periodic task appears automatically for the workers (`schedule`, `queue`).
+
+## 3. The building blocks
+
+### Request
+
+| field | purpose |
+|---|---|
+| `url`, `method`, `params`, `json`, `headers` | the HTTP request. Secrets go in `params` named `key`/`token`/`api_key`/`access_token`, or in `headers` – both are redacted from the archive and logs. |
+| `kind` | dispatch key for `parse()` |
+| `meta` | context for `parse()` (ids, names, depth); not part of the request identity |
+| `conditional` | send `If-None-Match`/`If-Modified-Since` from the archive (default on) |
+| `skip_unchanged` | when body hash and `parser_version` are unchanged, don't reload records – only follow-up requests are collected, so an interrupted run resumes (default on) |
+
+A request's identity (`fingerprint`) is method + url + params + json. The runner de-duplicates within a run.
+
+### Response
+
+`response.text`, `response.json()`, `response.soup` (BeautifulSoup, `html.parser`), `response.meta`
+(= `request.meta`), `response.url` (final URL, secrets redacted), `response.tdm_reserved`.
+
+### RunContext
+
+`ctx.params` (CLI `-p key=value`, worker parameters), `ctx.state` (persistent cursors, see below),
+`ctx.stats` (a Counter shown in `politrack status` – count what you skip and why), `ctx.log`.
+`ctx.conn` is available in `start()` only (e.g. to read which social accounts to poll); **`parse()` must
+not query the database** – that is what keeps it replayable and testable.
+
+### Records
+
+All in `politrack/core/records.py`:
+
+| record | use for |
+|---|---|
+| `PersonRecord`, `OrgRecord` | entity data from authoritative sources (names, photo, dates) |
+| `SocialAccount` | an account of a person/organisation (`verified=True` only if an official page links it) |
+| `VoteEvent`, `Event` | things with a key that claims point at (votes, sessions, trips) |
+| `Claim` (+ `facet`) | every fact. Facets: `Role`, `VotePosition`, `Money`, `Participation`, `Statement`, `Legal` |
+| `Document` | a document without a claim yet (headlines, press releases) – enrichers derive claims later |
+
+Claim types: `role vote donation money_flow statement appearance meeting travel side_job legal_proceeding
+lobbying media_mention career parliamentary_question government_decision wiki_edit declaration ownership
+affiliation party_finance other` (see [data-model.md](../data-model.md)).
+
+### Referring to people and organisations
+
+Never look up ids yourself. Describe what you know and let the Loader resolve it:
+
+```python
+PersonRef(ids={"parlament": "2339"})                                   # strong identifier (best)
+PersonRef(ids={"wikidata": "Q78869"}, name="Alexander Van der Bellen")
+PersonRef(name="Leonore Gewessler")                                    # exact alias match (titles stripped)
+PersonRef(hints={"surname_in_body": "nationalrat", "on": date(2024, 3, 1)}, name="Kickl")  # roll calls
+PersonRef(hints={"office": "Bundesministerin für Justiz", "on": date(2025, 6, 17)})          # office holder
+OrgRef(slug="oevp")                                                    # seeded organisations
+OrgRef(ids={"firmenbuch": "123456a"}, name="Muster GmbH", kind="company")  # created if unknown
+OrgRef(name="ÖVP", kind="party", create=False)                         # match only, never create
+```
+
+Identifier schemes: `parlament`, `wikidata`, `ep`, `uid`, `firmenbuch`, `rtr`, `eu_transparency`,
+`google_advertiser`, `facebook_page` – add a new scheme by simply using it (document it in data-model.md).
+Persons are only created by adapters with `creates_persons = True` (authoritative member lists); everyone
+else's claims about unknown people are skipped and counted as `unresolved_person`.
+
+Organisation names are matched case-, accent- and legal-form-insensitively (`normalize_org_name`), so
+"HYPO TIROL BANK AG" and "Hypo Tirol Bank Aktiengesellschaft" become one organisation.
+
+### Claim identity – facet or `key`
+
+The Loader computes a claim's identity from *resolved* entities and the facet: a donation is
+(kind, payer, payee, amount, date, period, reference), a vote position is (vote, person/party), a role is
+(person, body, role, start), a statement is (person/org, url), a participation is (event, person/org), a
+media mention is (url, person/org). So the same fact reported by two sources becomes **one claim with two
+pieces of evidence** – that is how corroboration works.
+
+Claims **without a facet** (side jobs, careers, declarations, holdings, questions …) need an explicit
+`key` – the Loader refuses them otherwise. Build it from what identifies the fact in the real world, not
+from your source's internals, so that another source could produce the same key:
+`key=("ministerrat", url, file_number, office)`, `key=("side_job", member_id, employer, function)`.
+A key always overrides the facet-based identity.
+
+### Summaries
+
+`summary_de` is required and shown to users; `summary_en` is optional but appreciated. Write neutral,
+factual one-liners ("X erhielt 2024 eine Spende von Y über 5.000 €"). Use `politrack.text.format_eur_de`
+for German amounts and `excerpt()` for quotes.
+
+### Sensitive claims
+
+Anything about criminal or legal matters: `claim_type="legal_proceeding"`, `sensitive=True`,
+`review=True` unless the source is an official court/parliament record. The UI then shows the
+presumption-of-innocence notice and the claim waits for human review.
+
+## 4. Incremental adapters and cursors
+
+For large or append-only sources, keep a cursor:
+
+```python
+def start(self, ctx):
+    since = ctx.params.get("since") or ctx.state.get("last_date") or "2024-01-01"
+    yield Request(url=..., params={"from": since}, kind="page")
+
+def parse(self, ctx, response):
+    ...
+    ctx.state.set("last_date", newest_date_seen)   # the only "write" parse() may do
+```
+
+`ctx.state` is persisted per adapter in the `state` table (and kept in memory during tests).
+
+## 5. Multi-publisher adapters
+
+Feeds of many outlets, or an aggregator that points at original publishers: define one `SourceSpec` per
+publisher (register extra specs with `register_spec`) and put `meta={"source": "<slug>"}` on each
+request – the Fetcher then applies *that* publisher's policy (store_fulltext, throttle) and evidence is
+attributed to it. See `sources/news.py`.
+
+## 6. Tests (required)
+
+1. Capture a fixture from the live source (one request, honouring robots.txt):
+   ```bash
+   politrack fixture musterland.members "https://www.landtag-musterland.at/api/members?period=current" list_current --kind list
+   ```
+2. Generate and **review** the golden output:
+   ```bash
+   UPDATE_GOLDEN=1 pytest tests/test_golden.py -k musterland
+   git diff tests/fixtures/musterland.members/
+   ```
+3. Unit-test tricky helpers (regexes, date formats) in `tests/test_parsers.py`.
+
+Keep fixtures small (cut lists to a few entries) and free of personal data beyond public roles.
+
+## 7. Changing an existing parser
+
+1. Change `parse()`, bump `parser_version`.
+2. `UPDATE_GOLDEN=1 pytest` and review the diff.
+3. After deployment: `politrack reparse <adapter>` replays the raw archive – no refetching. At the end of a
+   complete, error-free reparse, the adapter's claims that the new parser no longer produces are deleted
+   (unless another source backs them), so a fix that changes a claim's identity leaves no stale duplicate.
+   Two adapters sharing one `SourceSpec` should not emit the same claims, or a reparse of one removes the
+   other's until its next run.
+
+## 8. Checklist for the pull request
+
+- [ ] `SourceSpec` with honest `kind`, `trust`, `license`, `tos_notes` (legal checklist answered)
+- [ ] `parse()` is pure; counts skipped rows in `ctx.stats`
+- [ ] strong identifiers wherever the source has them
+- [ ] golden fixture + expected output, helper unit tests
+- [ ] entry in [docs/sources.md](../sources.md)
+- [ ] new claim types / identifier schemes documented in [data-model.md](../data-model.md) and the
+      frontend (`ClaimDetails.tsx`, messages)

docs/contributing/api.md

@@ -0,0 +1,45 @@
+# Extending the API (and the MCP server)
+
+Data flows out through three layers that are kept deliberately thin:
+
+```
+politrack/queries/<domain>.py   SQL, returns dicts          (shared by REST API and MCP server)
+politrack/api/schemas.py        Pydantic response models    (the contract → OpenAPI → TypeScript)
+politrack/api/main.py           FastAPI endpoints           (parameters, 404s, response_model)
+politrack/mcp_server.py         MCP tools for AI agents     (same query functions)
+```
+
+## Adding an endpoint, step by step
+
+1. **Query** – add a function to the matching module in `politrack/queries/` (people, organizations,
+   claims, votes, money, search, admin) and export it from `queries/__init__.py`.
+   * Only show visible claims: use `VISIBLE` (published, not merged) from `queries/common.py`.
+   * Respect the user's source filter: take a `SourceFilter` and use `evidence_json(source_sql)` to
+     aggregate evidence – claims whose evidence is entirely filtered out must disappear.
+   * Always parameterise (`%(name)s`), never format user input into SQL. `LIMIT/OFFSET` on lists.
+2. **Response model** – add a class to `api/schemas.py` (inherit from `Model`). Field names = column names.
+3. **Endpoint** – in `api/main.py`, with `response_model=...`. Reuse the dependencies `DB`, `Sources`,
+   `Limit`, `Offset`; comma-separated list parameters go through `csv_list()`.
+4. **Types for the web app**:
+   ```bash
+   cd backend && politrack openapi --ts ../web/src/lib/api-types.ts
+   ```
+   Then add a typed function to `web/src/lib/api.ts`. Commit the regenerated `api-types.ts`.
+5. **MCP tool** (if agents should use it) – a `@mcp.tool()` in `mcp_server.py` with a docstring that tells
+   the agent what the data means and how to cite it.
+6. **Test** – extend the API smoke test (see [testing.md](testing.md)).
+
+## Conventions
+
+* URLs: `/api/v1/<plural noun>[/<slug>[/<sub-resource>]]`; slugs, never internal ids, for people,
+  parties and organisations (vote events use ids – they have no natural slug).
+* Every claim-like object carries `evidence: [{url, title, source, published_at, quote, …}]` and
+  `source_count`.
+* Legal matters always include `legal_stage`; the client decides about the presumption-of-innocence
+  notice (`needsInnocenceNotice` in the web app, the `instructions` in the MCP server).
+* Admin endpoints require `Authorization: Bearer $POLITRACK_ADMIN_TOKEN` and are excluded from the schema.
+
+## Read-only database access for agents and BI tools
+
+The role `politrack_reader` (password `POLITRACK_READER_PASSWORD`) can `SELECT` everything. Useful views:
+`claim_with_evidence`, `person_status`, `person_identifier`. See [../data-model.md](../data-model.md).

docs/contributing/enrichers.md

@@ -0,0 +1,74 @@
+# Writing an enricher
+
+An **enricher** derives new facts from data that is already in the database: linking headlines to
+people, detecting trips and meetings, classifying legal matters, tagging topics, corroborating claims,
+computing embeddings. Adapters *collect*; enrichers *connect*.
+
+```python
+from collections.abc import Iterator
+
+from politrack.core import Enricher, RunContext, register
+from politrack.core.records import Claim, Evidence, PersonRef
+from politrack.core.spec import DERIVED
+
+
+@register
+class AmbassadorAppointments(Enricher):
+    spec = DERIVED                       # derived claims are attributed to "politrack" + the original evidence
+    name = "enrich.ambassadors"
+    description = "Ambassador appointments mentioned in Council of Ministers minutes"
+    schedule = "20 6 * * *"              # cron (UTC); runs on the `enrich` queue
+
+    def run(self, ctx: RunContext) -> Iterator[Claim]:
+        rows = ctx.conn.execute(
+            """SELECT d.id, d.url, d.title, d.published_at, d.full_text FROM document d
+               JOIN source s ON s.id = d.source_id
+               WHERE s.slug = 'bundesregierung' AND d.full_text ILIKE '%%Botschafter%%'
+                 AND NOT (d.meta ? 'ambassadors_at')"""
+        ).fetchall()
+        for doc in rows:
+            for name in find_appointees(doc["full_text"]):        # a pure, unit-tested helper
+                yield Claim(
+                    claim_type="career", person=PersonRef(name=name),
+                    summary_de=f"{name} als Botschafter:in vorgeschlagen",
+                    occurred_at=doc["published_at"],
+                    evidence=[Evidence(url=doc["url"], title=doc["title"], source="bundesregierung",
+                                       quote=..., doc_type="protocol")],
+                )
+            ctx.conn.execute("UPDATE document SET meta = meta || jsonb_build_object('ambassadors_at', now()) "
+                             "WHERE id = %s", (doc["id"],))
+            ctx.stats["documents"] += 1
+```
+
+## Rules
+
+* **Yield records, don't write facts yourself.** Records go through the same Loader as adapter output:
+  entity resolution, claim identity, evidence, corroboration status. Direct SQL is fine for *housekeeping*
+  (marking documents as processed, maintenance like closing roles at death – see `enrich/maintenance.py`).
+* **Cite the original.** Evidence of a derived claim points at the original document and names its
+  `source` slug, so the user's source filter and the corroboration logic work on the real publisher.
+* **Be incremental.** Remember what you processed (a marker in `document.meta`, or `ctx.state`), so a run
+  costs proportional to new data.
+* **Be conservative with people.** Enrichers never create persons (`creates_persons = False`); unknown
+  names are skipped and counted (`unresolved_person`). Prefer precision over recall – a missed link is
+  better than a wrong one.
+* **Sensitive output needs review.** Anything about legal matters: `sensitive=True, review=True`.
+* **Put logic in pure helpers** (regexes, classifiers) and unit-test them in `tests/test_extraction.py`.
+
+The runner commits every 200 records, so a crash loses little work.
+
+## Existing enrichers
+
+| name | what it does |
+|---|---|
+| `enrich.linker` | finds politicians (aliases, "Bundeskanzler Stocker" office titles) in headlines/teasers → `media_mention` claims |
+| `enrich.events` | trips, summits, bilateral meetings in official news → `Event` + participation, `travel`/`meeting` claims |
+| `enrich.legal` | investigation/indictment/verdict vocabulary in headlines → `legal_proceeding` claims (review queue) |
+| `enrich.topics` | keyword topics (mirrors Parliament's subject list) → `topic_assignment` |
+| `enrich.roles` | gives open-ended offices the holder's party; closes roles at death |
+| `enrich.rollcall` | faction majority positions derived from individual roll-call votes |
+| `enrich.corroborate` | merges near-duplicate claims, recomputes reported/corroborated/disputed |
+| `enrich.embed` | chunks + embeddings for semantic search/RAG (needs Ollama) |
+| `config.social_accounts` | imports `config/social_accounts.yaml` (curated accounts, party channels) |
+
+Run them all with `politrack enrich`, or one with `politrack run enrich.linker`.

docs/contributing/frontend.md

@@ -0,0 +1,81 @@
+# Extending the web app
+
+Next.js 15 (App Router, React Server Components) with next-intl (German default, English). Pages fetch
+from the API on the server; there is almost no client-side JavaScript.
+
+```
+web/
+  messages/de.json, en.json          all UI text (keys must exist in both)
+  src/app/[locale]/<route>/page.tsx  pages (German route names: /personen, /parteien, /geld …)
+  src/components/                    shared server components
+  src/lib/api.ts                     typed API client (server-side only; reads the filter cookie)
+  src/lib/api-types.ts               GENERATED from the backend's response models – do not edit
+  src/lib/format.ts                  dates, money, localized names
+  src/lib/prefs.ts                   the user's source/topic filter (cookie)
+```
+
+## Types come from the backend
+
+After any change to `backend/politrack/api/schemas.py`:
+
+```bash
+cd backend && politrack openapi --ts ../web/src/lib/api-types.ts
+cd ../web && npm run typecheck
+```
+
+`api.ts` re-exports the generated types (`Claim`, `MoneyFlow`, …) and adds one function per endpoint.
+`api.claims()`, `api.money()`, `api.social()` … automatically apply the visitor's source filter
+(`withPrefs`), so every list respects "which sources do I trust".
+
+## Adding a page
+
+1. Create `src/app/[locale]/<route>/page.tsx` – an async server component:
+   ```tsx
+   export default async function Page({params, searchParams}: Props) {
+     const {locale} = await params;
+     setRequestLocale(locale);
+     const t = await getTranslations('MyPage');
+     const rows = await api.something({...});
+     return <>…</>;
+   }
+   ```
+   Filters are plain `<form method="get">` – shareable URLs, no client state.
+2. Add the texts to **both** `messages/de.json` and `messages/en.json`.
+3. Link it from the navigation (`src/app/[locale]/layout.tsx`) or the home page's "Entdecken" cards.
+4. Page modules may only export Next.js page fields (`default`, `generateMetadata`, …); keep helpers and
+   constants module-local or in `src/components`.
+
+## Showing a new claim type
+
+Claims render through `ClaimList`; type-specific details come from the registry in
+`src/components/ClaimDetails.tsx`:
+
+```tsx
+const ambassador: Renderer = (claim, {t}) => (
+  <p className="claim-detail small">{t('country')}: {String(claim.data.country ?? '')}</p>
+);
+
+const DETAILS: Partial<Record<ClaimType, Renderer[]>> = {
+  …,
+  career: [counterpart, ambassador],
+};
+```
+
+Also add the label under `Claims.type.<claim_type>` in both message files, and (if it should appear) the
+type to the person page's `TABS`.
+
+## Non-negotiables in the UI
+
+* **Every fact shows its sources** (the "Quellen" disclosure under each claim, `[source]` links in
+  tables). Don't render a claim without its evidence.
+* **Presumption of innocence:** `needsInnocenceNotice()` in `ClaimList.tsx` decides; never remove the
+  notice for anything but an undisputed final conviction.
+* **Connections are not accusations:** pages that show networks or money of connected organisations
+  carry the note (`Network.note`, `Organizations.connectionNote`).
+* Unverified social accounts are marked (Wikidata-only handles).
+
+## Styling
+
+One global stylesheet (`src/app/globals.css`) with CSS variables for light/dark mode. Reuse the existing
+classes (`claims`, `claim-detail`, `chip`, `tag`, `notice`, `grid-2`, `compact`, `num`, `muted`, `small`)
+before adding new ones.

docs/contributing/testing.md

@@ -0,0 +1,50 @@
+# Testing
+
+```bash
+cd backend
+pytest                                   # everything below except the DB smoke test
+pytest tests/test_golden.py -k parlament # one adapter's golden tests
+UPDATE_GOLDEN=1 pytest tests/test_golden.py   # regenerate expected outputs (review the diff!)
+```
+
+| suite | what it covers | needs |
+|---|---|---|
+| `tests/test_core.py` | framework invariants: request identity, secret redaction, registry consistency, record serialisation, OpenAPI→TS generator, shared helpers | nothing |
+| `tests/test_parsers.py` | parsing helpers (vote texts, CSV layouts of the Rechnungshof, dates, normalisation) | nothing |
+| `tests/test_extraction.py` | roll-call lists, immunity reports, event/legal/office-title detection | nothing |
+| `tests/test_golden.py` | every captured fixture replayed through `adapter.parse()` and compared with `<case>.expected.json` | nothing |
+
+Parsers are pure, so none of these needs a database or network.
+
+## Golden fixtures
+
+```
+tests/fixtures/<adapter name>/<case>.request.json    {url, kind, meta, content_type, body_file}
+tests/fixtures/<adapter name>/<case>.<ext>           the captured body
+tests/fixtures/<adapter name>/<case>.expected.json   parse() output (records as JSON)
+```
+
+Capture with `politrack fixture <adapter> <url> <case> --kind <kind> --meta '{"…": …}'`. A fixture
+without `.expected.json` creates it on the first run and is skipped until you have reviewed it.
+
+When a source changes its format: capture a new case next to the old one (keep the old one if the
+publisher still serves old documents in that format), fix the parser, bump `parser_version`.
+
+## Against a real database
+
+The quickest end-to-end check uses the compose stack:
+
+```bash
+docker compose up -d db && docker compose run --rm migrate
+docker compose run --rm api politrack run parlament.persons --max-requests 20 -p periods='["XXVIII"]'
+docker compose run --rm api politrack enrich
+docker compose run --rm api politrack status
+curl -s localhost:8000/api/v1/stats
+```
+
+`--max-requests` keeps test runs polite. Each run is recorded in the `run` table (stats, errors) and
+shown by `politrack status` and `/api/v1/admin/runs`.
+
+## Frontend
+
+`npm run typecheck` (strict TypeScript against the generated API types) and `npm run build`.

docs/data-model.md

@@ -0,0 +1,115 @@
+# Data model reference
+
+Full DDL: [backend/migrations/0001_schema.sql](../backend/migrations/0001_schema.sql) (tables and columns
+carry `COMMENT`s). Seed data (chambers, parties, topics): `0002_seed.sql`. Design rationale:
+[architecture/01-design-from-scratch.md §3](architecture/01-design-from-scratch.md).
+
+## Overview
+
+```
+source ──< document >── story              publishers, the documents we saw (evidence), duplicate stories
+   │          │
+   │          └──< claim_evidence >──┐
+fetch_log (raw archive index)        │
+                                     claim ──< topic_assignment >── topic
+person ─┐                           ╱ │ ╲
+        ├── identifier, alias      ╱  │  ╲   one facet table per claim kind:
+organization ─┘                   ╱   │   ╲  role, vote_position(→vote_event), money_flow,
+social_account                         │      event_participant(→event), statement, legal_proceeding
+                                     chunk (text + embedding for RAG)
+run, state, app_user                 operations
+```
+
+## Entities
+
+| table | notes |
+|---|---|
+| `person` | name parts, birth/death date, photo. **No id columns** – external ids live in `identifier`. |
+| `organization` | `kind`: party, parliamentary_group, parliament, government, ministry, state_government, municipality, public_body, company, ngo, media, lobby, court, authority, eu_institution, other. `parent_id` for hierarchies. |
+| `identifier` | `(entity_type, entity_id, scheme, value)`, unique per (entity_type, scheme, value). |
+| `alias` | alternative names (with/without titles, short names, party abbreviations) used by the resolver and the linker. |
+| `social_account` | person *or* organisation, `platform`, `url`, `verified` (official source) and `source_slug`. |
+
+### Identifier schemes
+
+| scheme | entity | meaning |
+|---|---|---|
+| `parlament` | person | Parliament person id (PAD) – `parlament.gv.at/person/<id>` |
+| `wikidata` | person, org | Wikidata QID |
+| `ep` | person | European Parliament MEP id |
+| `firmenbuch` | org | Austrian company register number (FN), normalised `306527a` (`text.firmenbuch_number`) – from side-job declarations, procurement, party holdings |
+| `zvr` | org | Austrian association register number (ZVR) – from party reports (affiliated organisations) |
+| `uid` | org | VAT id (ATU…) |
+| `rtr` | org | legal entity id in the RTR media-transparency data |
+| `eu_transparency` | org | EU Transparency Register id |
+| `google_advertiser`, `facebook_page` | org | political advertiser ids in the ad libraries |
+| `politrack`, `politrack_org` | – | *virtual*: internal ids, used by enrichers in `PersonRef`/`OrgRef` (not stored) |
+
+## Claims
+
+`claim` holds the common fields: `claim_type`, subject (`person_id` and/or `organization_id`),
+counterpart (`counterpart_person_id`, `counterpart_org_id`), time (`occurred_at` or `valid_from`/`valid_to`),
+`summary_de`/`summary_en`, `data` (JSONB, type-specific extras), `status`, `sensitive`, `review_status`,
+`merged_into`, `fingerprint` (identity), `created_by` (adapter/enricher name).
+
+| claim_type | facet table | typical source |
+|---|---|---|
+| `role` | `role` (role, level, body, party, region, start/end) | parliament, Wikidata, EP |
+| `vote` | `vote_position` → `vote_event` | parliament (faction votes, roll calls), `enrich.rollcall` |
+| `donation` | `money_flow` | Rechnungshof donation lists, party reports |
+| `money_flow` | `money_flow` | procurement, RTR media transparency, party funding, political ads |
+| `side_job` | – (`data`, counterpart org) | parliament declarations |
+| `statement` | `statement` (context, platform, url, excerpt) | speeches, social posts |
+| `appearance`, `meeting`, `travel` | `event_participant` → `event` | calendar, hearings, Council of Ministers, `enrich.events`, EC meetings |
+| `legal_proceeding` | `legal_proceeding` (stage, offense, authority) | immunity reports, `enrich.legal` (reviewed) |
+| `lobbying` | – | EC meetings with interest representatives |
+| `media_mention` | – | `enrich.linker` on news headlines |
+| `career` | – | Wikidata employers/positions |
+| `parliamentary_question` | – | parliament |
+| `government_decision` | – | Council of Ministers minutes |
+| `wiki_edit` | – | Wikipedia edits from institutional IP ranges (subject = network owner) |
+| `declaration` | – | EP declarations of interests |
+| `ownership`, `affiliation` | – | party reports (holdings, affiliated organisations) |
+| `party_finance` | – | party annual reports (totals in `data`) |
+
+**Status:** `reported` (one source) → `corroborated` (≥ 2 distinct sources support/mention it) or
+`disputed` (a source contradicts). Computed by the Loader on every new piece of evidence and by
+`enrich.corroborate`.
+**Review:** `review_status` = `pending` for claims that need a human (legal matters derived from
+headlines); only `published` claims are served.
+**Merging:** near-duplicates are merged by `enrich.corroborate` (`merged_into`), never deleted.
+
+### Claim identity (fingerprint)
+
+| facet | identity |
+|---|---|
+| explicit `key` | `(claim_type, "key", *key)` |
+| Role | person, body, role, start |
+| VotePosition | vote event, person/party |
+| Money | kind, payer, payee, amount, paid_on, period, reference |
+| Statement | person/org, url |
+| Participation | event, person/org |
+| media_mention | url, person/org |
+
+Claims without a facet must carry a `key` (see [contributing/adapters.md](contributing/adapters.md)).
+
+## Evidence
+
+`document` = one URL we saw (title, published_at, excerpt, `full_text` only if the publisher's terms
+allow it and no TDM reservation applies). `claim_evidence` links a claim to a document with `quote`,
+`locator`, `stance` (supports/contradicts/mentions) and `confidence`. `story` groups the same story
+across outlets (independence for corroboration).
+
+## Views
+
+| view | purpose |
+|---|---|
+| `person_status` | active / former / party_member / deceased per person, `verified` (current office confirmed by parliament or EP data), `last_office_end`, `current_party_ids` |
+| `person_identifier` | person with its main identifiers as columns (parlament, wikidata, ep) |
+| `claim_with_evidence` | published, unmerged claims with an `evidence` JSON array – the easiest starting point for agents and SQL users |
+
+## Operations tables
+
+`fetch_log` (every HTTP fetch: request, status, sha256, stored body, parse state), `run` (every adapter /
+enricher run with stats and errors), `state` (adapter cursors), `chunk` (text chunks + pgvector
+embeddings), `app_user` (reviewers).

docs/deployment-unraid.md

@@ -0,0 +1,114 @@
+# Deployment on Unraid
+
+## Prerequisites
+
+* Unraid 6.12+ with the **Docker Compose Manager** plugin (Community Applications).
+* An appdata share, for example `/mnt/user/appdata/politrack`.
+* Network access from the server to PyPI/npm for the image builds (`files.pythonhosted.org`,
+  `registry.npmjs.org`) – see "Troubleshooting".
+* Optional: a reverse proxy (SWAG, Nginx Proxy Manager, Traefik or Caddy) and SSO (Authentik or Authelia)
+  if the site is exposed; an Ollama container for semantic search.
+
+## Steps
+
+1. Clone the repository onto the server (for example `/mnt/user/appdata/politrack/src`) or add it as a
+   stack in Compose Manager pointing at this folder.
+2. `cp .env.example .env` and set at least:
+   * `DATA_DIR=/mnt/user/appdata/politrack` – database (`postgres/`) and raw archive (`raw/`) live here
+   * `POSTGRES_PASSWORD`, `POLITRACK_READER_PASSWORD`
+   * `POLITRACK_USER_AGENT` with a real contact address (publishers must be able to reach you)
+   * optional: `POLITRACK_ADMIN_TOKEN`, `POLITRACK_OLLAMA_URL`, API keys (YouTube, Meta Ad Library, X)
+3. Build and start:
+   ```bash
+   docker compose build
+   docker compose up -d
+   ```
+   `migrate` runs once (schema, seed data, source registry, job-queue tables), then `api`, `web`, `mcp`
+   and three workers start:
+
+   | service | queues | what runs there |
+   |---|---|---|
+   | `worker-scrape` | scrape, social | most adapters (hourly to daily) |
+   | `worker-bulk` | bulk | large or slow downloads: procurement, RTR, ad libraries, EC meetings, GDELT |
+   | `worker-enrich` | enrich | linking, events, legal detection, topics, corroboration, embeddings |
+
+4. Initial load (otherwise the schedules pick everything up over the next day):
+   ```bash
+   docker compose exec api politrack sources list            # what exists, when it runs
+   docker compose exec api politrack run parlament.persons   # members first: other adapters resolve against them
+   docker compose exec api politrack run wikidata.politicians
+   docker compose exec api politrack run europarl.meps
+   docker compose exec api politrack run parlament.items     # votes etc. – the longest run (30-60 min)
+   docker compose exec api politrack run config.social_accounts
+   docker compose exec api politrack enrich
+   ```
+5. Open `http://<unraid-ip>:3000` (web), `http://<unraid-ip>:8000/api/docs` (API) and
+   `http://<unraid-ip>:8001/mcp` (MCP).
+
+**Upgrading from the first version (v1):** the schema was rewritten (`0001_schema.sql`). Start with a
+fresh database: stop the stack, move `${DATA_DIR}/postgres` away, `docker compose up -d`, re-run the
+initial load. Nothing is lost that the sources can't provide again.
+
+## Scaling and separate deployments
+
+* More throughput: `docker compose up -d --scale worker-scrape=3`. Queueing locks ensure one adapter
+  never runs twice at the same time; per-host throttling is per process, so keep one replica per queue
+  if you are close to a publisher's rate limit.
+* Another machine (different egress IP, GPU): run the same image with `POLITRACK_DATABASE_URL` pointing at
+  this database and `procrastinate --app=politrack.worker.app worker --queues=<queues>`. Mount a raw
+  archive volume there too (or set `POLITRACK_RAW_DIR` to shared storage).
+
+## Semantic search (optional)
+
+```bash
+docker compose --profile ai up -d ollama
+docker compose exec ollama ollama pull bge-m3
+# .env: POLITRACK_OLLAMA_URL=http://ollama:11434
+docker compose up -d
+```
+
+bge-m3 runs fine on CPU for this data volume. With an NVIDIA GPU, add a device reservation
+(`deploy.resources.reservations.devices`) to the ollama service.
+
+## Backups
+
+* Database: a nightly `pg_dump`, e.g. with the **User Scripts** plugin:
+  ```bash
+  docker exec politrack-db-1 pg_dump -U politrack -Fc politrack > /mnt/user/backups/politrack/$(date +%F).dump
+  find /mnt/user/backups/politrack -mtime +30 -delete
+  ```
+* Raw archive (`${DATA_DIR}/raw`): optional to back up – it saves refetching after parser fixes, and it can
+  be pruned (`find raw -mtime +365 -delete`); missing blobs are simply refetched.
+
+## Exposing publicly
+
+* Expose only `web` (and optionally `api`) through the reverse proxy. Keep `mcp` and the DB port LAN- or
+  VPN-only, or protect them with SSO or an API key at the proxy.
+* Read [legal.md](legal.md) before going public (Impressum, privacy policy, §7b MedienG).
+
+## Monitoring
+
+* `docker compose exec api politrack status` – recent runs with stats and errors (table `run`).
+* `GET /api/v1/admin/runs` with `Authorization: Bearer $POLITRACK_ADMIN_TOKEN`.
+* Fetch log: `SELECT adapter, status, count(*) FROM fetch_log WHERE fetched_at > now() - interval '1 day' GROUP BY 1, 2;`
+* Job queue: `SELECT task_name, status, count(*) FROM procrastinate_jobs GROUP BY 1, 2;`
+* Uptime Kuma can poll `http://api:8000/healthz`.
+
+## Troubleshooting
+
+**`pip install` / `npm install` fail during the build with TLS errors, while other sites work.**
+Something between the server and the internet blocks the package CDNs by host name (TLS SNI): typically
+a DNS filter (Pi-hole/AdGuard, router "parental control"/"web protection"), a firewall with TLS
+inspection, or antivirus web protection on the build machine. Check:
+
+```bash
+nslookup files.pythonhosted.org                 # resolves to a Fastly address, not 0.0.0.0 / your router?
+curl -v https://pypi.org/simple/ -o /dev/null   # works?
+curl -v https://files.pythonhosted.org/ -o /dev/null
+curl -v https://registry.npmjs.org/ -o /dev/null
+```
+
+If pypi.org works but files.pythonhosted.org / registry.npmjs.org fail during the TLS handshake, allow
+these host names in the filter/firewall/AV (also `*.pythonhosted.org`, `registry.yarnpkg.com` if used). On
+Windows, the System event log (source *Schannel*) and the AV's web-protection log show which component
+reset the connection.

docs/investigative-data.md

@@ -0,0 +1,69 @@
+# More data points for investigative work
+
+A shortlist of additional **legally accessible** sources and the correlations they enable, ranked by
+value per effort. "Status" refers to this project: ✅ done · 🟡 next · ⛔ not accessible · 🔍 manual research only
+(no bulk access).
+
+## 1. Money that follows power
+
+| Data | Source & access | Why it matters | Status |
+|---|---|---|---|
+| **Public contracts** ≥ €50k (core data, *Kerndaten*) | Contracting authorities must publish award metadata as open data on data.gv.at within 30 days (§62 BVergG 2018; amended from 1 Oct 2026, check the new version) | Correlate **donors ↔ contract winners ↔ awarding ministry/party**. This is the classic pay-to-play pattern | ✅ `vergabe.awards` |
+| Government **advertising** per ministry → outlet | RTR Medientransparenz open data | Which ministries fund which media, and does coverage change? Inserate affairs | ✅ `rtr.medientransparenz` (ads and media subsidies) |
+| Party **annual reports** (*Rechenschaftsberichte*) and audit results | Rechnungshof (PDF) | Income, related organisations (*Vorfeldorganisationen*), sponsoring, loans; discrepancies the Rechnungshof finds | ✅ `rechnungshof.party_reports` (XLSX totals, loans, holdings, affiliated organisations; audit findings 🟡) |
+| Public **party funding** (*Parteienförderung, Klubförderung, Akademieförderung*) | Federal/state budgets, Rechnungshof | Context for donations; how much taxpayers pay each party | ✅ from the annual reports (federal budgets 🟡) |
+| **EU agricultural subsidies** per recipient | EU-mandated transparency database (AMA) | Farmer-politicians and their associations | 🟡 |
+| COVID aid (COFAG) large recipients | Published lists | Companies close to parties that received aid | 🟡 |
+| Political **ad spending** online | Meta Ad Library API (verified developer account), Google Political Ads Transparency (public BigQuery dataset) | Spend per party/politician, targeting, timing before elections | ✅ `ads.google` (public bundle), `ads.meta` (token) |
+| Company roles (management, supervisory boards, shareholdings) | Firmenbuch (fee per query; the status of free EU "high-value dataset" access in Austria is unclear) | Revolving door, conflicts of interest, donor networks | 🔍 (until free bulk access is confirmed) |
+| Land register (*Grundbuch*) | Fee per query | Property deals involving politicians | 🔍 |
+
+## 2. Influence & access
+
+| Data | Source | Why | Status |
+|---|---|---|---|
+| Lobbying register | BMJ (public part) | Who lobbies, for which clients | ⛔ Lotus Notes application without stable URLs – not reliably accessible |
+| **EU Transparency Register** and Commission meetings | Open data (also via LobbyFacts.eu) | Meetings of the Austrian commissioner/cabinet and of MEPs (MEPs must publish meetings when they are rapporteurs) | ✅ Commission meetings `eu.commission_meetings`; MEP meetings 🟡 |
+| MEP declarations of financial interests | European Parliament website | Side income of Austrian MEPs (an EU-level counterpart to the Unv-Transparenz lists) | ✅ `europarl.meps` |
+| Cabinet staff (*Kabinettsmitarbeiter*), appointments to state-owned companies (ÖBAG holdings, ORF bodies, Nationalbank) | Answers to parliamentary questions (already ingested as claims), annual reports, Wikidata | Patronage (*Postenschacher*) | 🟡 (extract from answers) |
+| Council of Ministers minutes (*Ministerratsprotokolle*) | Bundeskanzleramt | What the government decided, and when | ✅ `bundesregierung.ministerrat` |
+| EU Council participant lists | Council website (PDF) | Ministers' travel and attendance | 🟡 |
+
+## 3. Legal & accountability
+
+| Data | Source | Why | Status |
+|---|---|---|---|
+| Immunity waiver requests | Parliament | Investigations against MPs: authority, alleged offence, outcome | ✅ |
+| Prosecutor and court **press releases** (WKStA, Landesgerichte' media offices), media dates of prominent trials | justiz.gv.at | Official statements on stages of high-profile cases. Feed them into `legal_proceeding` with *official* evidence (no headline guessing) | ✅ WKStA `justiz.wksta` (documents; claims after review), courts 🟡 |
+| Court decisions | RIS (anonymised) | Link only when a case number is cited in an official source | 🟡 |
+| **Investigative committees** (*Untersuchungsausschüsse*): witness lists, protocols, final reports | Parliament open data | Who testified, what they said, contradictions with later statements | ✅ hearings (who, when); protocols 🟡 |
+| Audit reports on ministries and agencies | Rechnungshof | Mismanagement in departments led by a given minister | 🟡 |
+| Insolvency notices (Ediktsdatei) | Justiz | Companies tied to politicians. Check the terms before automating | 🔍 |
+
+## 4. Words vs. deeds
+
+| Idea | Data | Status |
+|---|---|---|
+| **Say vs. vote**: compare statements (speeches, posts, interviews) with roll-call and faction votes on the same topic | statements + votes + embeddings | ✅ basic: votes show whether the person spoke on the item; topic-level comparison 🟡 |
+| Election programmes and the government programme vs. what was passed | Party programmes (Manifesto Project: check its licence), government programme PDF, parliament items | 🟡 |
+| Dissent tracker | Roll-call votes (✅): who breaks with their faction, and on which topics | ✅ basic |
+| Wikipedia edits from parliament/government IP ranges | Wikipedia API (open) + public IP allocations | ✅ `wikipedia.institution_edits` |
+
+## 5. Coverage & reach
+
+| Data | Source | Status |
+|---|---|---|
+| Candidate lists of all elections (federal, state, municipal) | BMI / state election authorities, official gazettes | 🟡. This is the most reliable way to find *local* politicians and party affiliations |
+| Election results per municipality | BMI open data | 🟡 |
+| TV appearances (Pressestunde, ZIB 2, Im Zentrum guest lists) | ORF programme metadata / headlines | ✅ headline detection, 🟡 programme metadata |
+| Bluesky, Mastodon, public Telegram channels, YouTube; X via the official API | open APIs / public web preview / paid API | ✅ (Threads 🟡) |
+
+## Correlations the data model already supports
+
+* **Person timeline** (all claims over time, filter by period): ✅
+* **Calendar**: sessions, committees, trips, meetings, per-day activity: ✅
+* **Former politicians**: latest coverage plus career entries (revolving door): ✅
+* **Corroboration**: the same fact from independent sources is merged and shown as confirmed: ✅
+* **Follow the money**: per person, the organisations they are connected to (side jobs, careers, party, meetings) and those organisations' donations, contracts, ads and funding: ✅ (`/personen/<slug>?tab=money`, `/api/v1/persons/<slug>/network`)
+* **Organisation pages**: money paid/received by kind, top counterparties, connected people: ✅
+* Next: a visual **network graph** and company roles (Firmenbuch – no free bulk access yet).

docs/legal.md

@@ -0,0 +1,41 @@
+# Legal & ethical guard-rails
+
+> This is an engineering summary, **not legal advice**. Before publishing the site
+> publicly, have an Austrian media/data-protection lawyer review it. Partner NGOs such as
+> epicenter.works, Forum Informationsfreiheit and MeineAbgeordneten can help.
+
+## 1. Collecting data (copyright, database rights, ToS)
+
+* **Official works** (laws, official decrees and publications, §7 UrhG) are not protected by copyright. Parliament open data is CC BY 4.0 and needs attribution (shown on the method page and in every evidence link).
+* **Text & data mining** (§42h UrhG / Art. 4 DSM Directive) is allowed for lawfully accessible content **unless the rightholder reserved it in machine-readable form**. The Fetcher detects TDMRep reservations for every origin (no text and no raw archive is kept for reserved sites), and media content is limited to headline, teaser and link (see [sources.md](sources.md)).
+* **Database right** (§76c ff. UrhG): systematically extracting substantial parts of someone else's curated database (for example MeineAbgeordneten) is not allowed without permission, so we ask for partnerships instead.
+* **ToS**: we never scrape platforms whose terms prohibit it (X, Meta, most newspapers). We use official APIs or leave the source out. Each source's assessment is recorded in its `SourceSpec.tos_notes` (shown on the method page and in `/api/v1/sources`).
+* **Raw archive**: responses are archived only where the publisher's terms allow it (`SourceSpec.keep_raw`) and never for TDM-reserved sites. The archive serves re-parsing and auditability, not republication.
+* **X API**: the Developer Agreement requires removing content of posts that were deleted on X. `social.x` keeps no raw archive or full text, but the deletion sync is still open – don't enable it in production before it exists (see roadmap).
+* **Messaging platforms**: only public Telegram broadcast channels through Telegram's public web preview. No WhatsApp (no public read access, terms forbid automated collection), no Signal, no private groups.
+
+## 2. Personal data (GDPR / DSG)
+
+* Politicians are public figures, but this is still personal data. Our legal basis is **legitimate interest / public interest in political transparency** (Art. 6(1)(f) GDPR). Freedom of expression and information (Art. 85 GDPR) supports this, but Austria's implementation of the media privilege (§9 DSG) was challenged before the Constitutional Court and amended. **Check the current version.**
+* **Scope:** only the *public role* of a person: offices, votes, statements, public appearances, declared side jobs, donations to parties, and public court matters. No private addresses, family members (unless they hold office themselves) or health data.
+* **Data on criminal offences** (Art. 10 GDPR) is especially restricted. We **only republish what official bodies or reputable media have already made public**, always with the procedural stage and the presumption of innocence, and never infer anything ourselves.
+* **Private donors** named in the Rechnungshof lists: republishing information the law requires to be published is defensible, but show it exactly as published, link to the source, and honour erasure requests if the official publication is withdrawn. Private donors stay plain names (no person entity, no address – the record types have no address field); only companies and associations become organisations.
+* **Wikipedia edits from institutional IP ranges** are attributed to the *network owner* (parliament, government), never to a person.
+* **Connections are not accusations**: network and "follow the money" views say so explicitly; a side job or meeting is shown as what it is.
+* Provide a **privacy policy**, a **contact for data-subject requests** (access, rectification, objection) and a documented **balancing test** (LIA).
+
+## 3. Media law (if the site is public)
+
+* **Impressum/Offenlegung** (§§24, 25 MedienG) is required for a website providing information.
+* **§7b MedienG (presumption of innocence)**: anyone not finally convicted must not be presented as guilty. Implemented rule: every `legal_proceeding` claim shows **"Es gilt die Unschuldsvermutung."** unless its stage is `convicted_final` *and* it is not `disputed`. Acquittals, dropped proceedings, diversion and immunity requests keep the notice, since they confirm there is no finding of guilt. The MCP instructions tell agents the same. Summaries are phrased neutrally and attributed ("Ermittlungen (laut Bericht)", "nicht rechtskräftig verurteilt").
+* **Review before publication**: legal matters detected automatically in headlines are stored with `review_status = 'pending'` and are **not shown** until a person approves them (`politrack review`, or `POST /api/v1/review/{id}` with `POLITRACK_ADMIN_TOKEN`). Official data (Parliament immunity requests) is published directly, because the source itself is the official record.
+* **Defamation** (§111 StGB, §1330 ABGB): only publish what a source states, attributed to that source ("laut DER STANDARD …"), with a link. Disputed claims are labelled as such.
+* **Right of reply / corrections:** offer a corrections form so politicians can submit counter-evidence. Counter-evidence becomes `stance = 'contradicts'` evidence, which marks the claim `disputed` (it is never silently deleted).
+
+## 4. Operational rules
+
+* Identify the crawler honestly (User-Agent with contact address) and respond to abuse mails.
+* Keep raw HTTP fetches minimal and do not mirror third-party content.
+* Keep an audit trail: every claim records `created_by` (which scraper or moderator) and when.
+* Keep humans in the loop for anything `sensitive` (legal matters): automatically detected claims wait in the review queue (`review_status = 'pending'`) until a moderator approves them.
+* Every run is logged (`run` table) and every fetch too (`fetch_log`: URL with secrets redacted, status, hash) – useful when a publisher asks what was accessed.

docs/market-research.md

@@ -0,0 +1,38 @@
+# Market research: existing politician-tracking services
+
+Researched October 2026. The goal was to find what already exists for Austria,
+what comparable projects in other countries do well, and where the gaps are.
+
+## Austria
+
+| Project | What it does | Gap / relevance for us |
+|---|---|---|
+| [MeineAbgeordneten.at](https://www.meineabgeordneten.at) | NGO (Verein für Transparenz und Demokratie, founded 2011 by Respekt.net). Around 800 hand-curated dossiers covering federal and EU politicians and members of state governments and parliaments: functions, education, side income, association roles, APA-OTS releases ([presentation 2024](https://politik-lernen.at/dl/nrtoJKJKKokoNJqx4KJK/Pr_sentation_MeineAbgeordneten_30_4_24_pdf), [overview](https://politik-lernen.at/meineabgeordneten_30minutenplus)). | Closest existing service. Its dossiers are curated by hand and do not track votes, media appearances or donations on an ongoing basis. It is also the only Austrian source listed by OpenSanctions. **Best partner rather than competitor**: ask for a data partnership. |
+| [Parteispenden.at](https://www.opengovpartnership.org/stories/who-funds-political-parties-what-we-have-learned-from-an-effort-to-create-meaningful-transparency/) (Forum Informationsfreiheit) | Aggregates party-financing information with visualisations. | Covers donations only. Its authors note that donor names alone don't explain which interests sit behind them. We should link donors to companies and lobby registrations. |
+| [Parlament.gv.at Open Data](https://www.parlament.gv.at/recherchieren/open-data/) | 25 official datasets (CC BY 4.0) with a JSON API: MPs since 1918, items, resolutions, stenographic protocols, committee memberships, the Parlamentskorrespondenz and more. | This is our backbone. The website does show faction votes, but there is no cross-source view. |
+| [OffenesParlament.at](https://offenesparlament.readthedocs.io/) | Open-source parliament data platform from around 2017 with search subscriptions. | Looks dormant. Shows there is interest, but it covered parliament only. |
+| [OpenSanctions – Austria](https://www.opensanctions.org/countries/at/) | About 3,100 Austrian PEPs, built mostly from Wikidata and MeineAbgeordneten. | Useful for cross-checking and IDs. Licence is CC BY-NC (non-commercial only). |
+| wahlkabine.at and other voting-advice apps | Compare party positions before elections. | Show positions, not actions. |
+
+**Finding:** no Austrian service brings together *votes + money + appearances + statements +
+legal matters* with **per-claim citations** and **user-selectable source trust**.
+Local politics (Landtage, municipalities) is almost entirely uncovered.
+
+## Other countries (ideas worth copying)
+
+| Project | Country | Idea to adopt |
+|---|---|---|
+| [abgeordnetenwatch.de](https://www.abgeordnetenwatch.de/api) | DE | Clean open **API (CC0)** with roll-call votes, side jobs and Q&A. Fair-use limits are documented (30 req/min, bulk downloads at night). We publish our API in the same way. |
+| [Real-O-Mat](https://netzpolitik.org/2025/real-o-mat-taten-zaehlen-mehr-als-worte/) (FragDenStaat) | DE | A "voting-advice app based on *actual votes*". Our faction votes make an Austrian version possible. |
+| [HowTheyVote.eu](https://howtheyvote.eu/about) | EU | Roll-call votes of the European Parliament, open [CSV data](https://github.com/HowTheyVote/data), results within about an hour. We use it for Austrian MEPs. |
+| [Abgeordnetenwatch MCP](https://glama.ai/mcp/servers/t5pjdt7p1a) / UK Parliament MCP | DE/UK | Parliament data exposed to AI agents via MCP. **We ship an MCP server too.** |
+| TheyWorkForYou (mySociety), GovTrack, OpenSecrets | UK/US | Mature examples: alert subscriptions, speech search, money-in-politics views. |
+| LobbyFacts / Integrity Watch EU | EU | Meetings between commissioners and lobbyists. Austria has no equivalent register of ministers' meetings, which is a gap to note. |
+
+## Consequences for the design
+
+1. **Build on official open data first** (Parliament, Rechnungshof, RTR, RIS, EP). This is legally clean and high-trust.
+2. **Treat everything as claim + evidence**, so several sources can confirm or contradict the same fact (no project above does this generically).
+3. **Partner, don't scrape** curated NGO databases (MeineAbgeordneten, Parteispenden.at).
+4. **Expose an API and an MCP server** so journalists, researchers and AI agents can reuse the data.
+5. **Local level** is the biggest gap. Start with Wikidata (Landtage are well covered, mayors only partly) and add per-state sources over time.

docs/roadmap.md

@@ -0,0 +1,75 @@
+# Roadmap
+
+## Done
+
+**Foundation (v1)** – claim/evidence model, source trust and topic filters, parliament (members,
+mandates, functions, side jobs, speeches, faction votes, roll calls, immunity waivers, questions),
+Wikidata (state/local offices, party membership, careers), Rechnungshof donations, news feeds and
+sitemaps (headline-level, TDM-aware), entity linking, travel/meeting extraction, legal-matter detection
+with review, corroboration, activity status, former politicians, calendar, timeline, REST API, MCP server,
+Next.js frontend (de/en), Docker Compose for Unraid.
+
+**Re-architecture (v2)** – see [architecture/02-comparison-and-decisions.md](architecture/02-comparison-and-decisions.md):
+- [x] Adapter framework: `SourceSpec` in code, pure `parse()`, Fetcher with robots/TDM/throttle/conditional GET
+- [x] Raw archive + `politrack reparse`; change detection (body hash + parser version)
+- [x] Single writer (Loader), identity from resolved entities, `identifier` table, resolution hints
+      (roll-call surnames, office holders on a date), legal-form-insensitive organisation matching
+- [x] Registry + entry points; periodic tasks generated from adapter metadata; queues scrape/social/bulk/enrich
+- [x] Typed API (Pydantic) → OpenAPI → generated TypeScript; golden-fixture tests
+- [x] Contributor documentation
+
+**New data (v2)**:
+- [x] Rechnungshof annual party reports (totals, public funding, loans, holdings, affiliated organisations)
+- [x] Public procurement core data (contract awards, eForms)
+- [x] RTR media transparency (government ads, media subsidies)
+- [x] Council of Ministers minutes (agenda items per reporting minister)
+- [x] U-committee hearings (who testified when)
+- [x] WKStA press releases (documents for search; legal claims only after review)
+- [x] European Parliament: Austrian MEPs, declarations of interests
+- [x] European Commission meetings with interest representatives
+- [x] Wikipedia edits from parliament/government IP ranges
+- [x] Political ads: Google (bundle), Meta Ad Library (opt-in token)
+- [x] Social: Bluesky, Mastodon, public Telegram channels, YouTube (key), X via official API (token);
+      party/institution channels; curated `social_accounts.yaml`
+- [x] Pages: organisations, money explorer, social feed, Wikipedia edits, person network ("follow the money"),
+      say-vs-vote on roll calls
+
+## Next
+
+**Data**
+- [ ] X deletion sync (Developer Agreement: remove posts deleted on X) – required before enabling `social.x` in production
+- [ ] Full speech texts and interjections from the protocols; committee memberships (list 250)
+- [ ] HowTheyVote.eu / EP roll-call votes of Austrian MEPs
+- [ ] EU Council participant lists, ministries' press releases, Federal President's schedule
+- [ ] Election results and candidate lists; AMA/TAM subsidy recipients (after checking bulk access and terms)
+- [ ] Cabinet staff and appointments to state-owned companies (from answers to parliamentary questions)
+- [ ] One adapter per Landtag (OGD where available); municipal councils and mayors
+- [ ] Lobbying register – if the BMJ ever offers stable URLs or an export
+
+**Quality**
+- [ ] Golden fixtures for every adapter (the framework and first cases are in place)
+- [ ] Better event dates (explicit dates in the text instead of publication dates)
+- [ ] Entity resolution for donors/companies via Firmenbuch numbers from procurement notices
+- [ ] Wayback Machine archiving of evidence URLs (`document.archive_url`)
+
+**Product**
+- [ ] Community link submissions and a corrections/right-of-reply form
+- [ ] Alerts (e-mail/RSS) for followed politicians
+- [ ] Optional accounts via reverse-proxy SSO using `app_user`
+- [ ] Partnerships: MeineAbgeordneten, Parteispenden.at, APA (OTS full-text licence)
+- [ ] Map view of trips and appearances
+
+## Known limitations
+
+- Votes in the Nationalrat are faction-level except roll calls (an Austrian parliamentary practice).
+- Split votes (factions voting differently on parts of a bill) keep the official text but no faction
+  positions, because the summary cannot be attributed reliably.
+- Activity status: parliament and EP data are authoritative (`status_verified`). Open-ended roles from
+  Wikidata count as current for 15 years after their start, so some state/local politicians may show
+  wrong status until per-state adapters exist.
+- Roll-call surnames that are ambiguous without a link in the protocol are skipped, not guessed.
+- Travel/meeting extraction is rule-based; dates are publication dates of the source.
+- News coverage is headline-level; mentions are linked by exact name or "office title + surname".
+- Telegram: only channels with the public web preview; only the latest posts per run.
+- Council of Ministers items are attributed to the office holder on that date; if the resolver cannot
+  find who held the office (missing role data), the item is skipped and counted.

docs/sources.md

@@ -0,0 +1,158 @@
+# Source catalogue & access policy
+
+**Rule #1: we only collect data in ways the publisher permits.** We prefer official APIs and open-data
+downloads, then feeds and sitemaps that publishers offer for automated discovery, then polite crawling of
+public government pages. We never log in, never bypass paywalls or bot protection, and never scrape
+platforms whose ToS forbid it. The supermarket price-tracker case from the brief failed for exactly that
+reason: it scraped commercial sites against their ToS.
+
+Enforcement is technical, not a convention – every request goes through the Fetcher
+([`backend/politrack/core/fetch.py`](../backend/politrack/core/fetch.py)):
+
+* an honest User-Agent with a contact address (`POLITRACK_USER_AGENT`),
+* robots.txt is obeyed (cached per origin),
+* per-host throttling (`SourceSpec.min_interval`, raised by a robots.txt `Crawl-delay`), `Retry-After` honoured,
+* TDM reservations are detected (W3C TDMRep `/.well-known/tdmrep.json`, `/tdmrep.json`, `tdm-reservation`
+  header). The text-and-data-mining exception (Art. 4 DSM Directive, §42h UrhG) does not apply once a site
+  reserves its rights in machine-readable form; we then keep headline and link only,
+* per-publisher storage rules from the `SourceSpec`: `store_fulltext` (may we keep text?) and `keep_raw`
+  (may we archive the raw response?).
+
+**robots.txt exemptions** (`SourceSpec.robots_exempt`) exist only for documented APIs whose operators allow
+programmatic use but list the endpoint in robots.txt to keep search-engine crawlers away:
+the Wikidata Query Service (`/sparql`; [Wikimedia User-Agent policy](https://meta.wikimedia.org/wiki/User-Agent_policy)),
+the MediaWiki API of de/en Wikipedia (`/w/api.php`; [API etiquette](https://www.mediawiki.org/wiki/API:Etiquette)),
+and the authenticated X API (governed by the developer agreement, not by crawler rules).
+
+Status: ✅ implemented (adapter name in `code`) · 🟡 planned · ⛔ deliberately not used.
+
+## Official sources (high trust)
+
+| Source | Data | Access | Licence / terms | Status |
+|---|---|---|---|---|
+| Parlament Österreich | MPs since 1918, mandates, functions, **side jobs & income category** (§6 Unv-Transparenz-G), social-media links, plenary speeches, **faction votes** per item and reading | JSON API (`/Filter/api/filter/data/409,101,251`, `?json=TRUE` detail pages) | [CC BY 4.0](https://www.parlament.gv.at/recherchieren/open-data/daten-und-lizenz/) | ✅ `parlament.persons`, `parlament.items` |
+| – stenographic protocols | **Roll-call votes**: every member's yes/no from the "Mit „Ja“/„Nein“ stimmten …" lists | HTML fragments | CC BY 4.0 | ✅ `parlament.items` |
+| – immunity waivers | prosecutors' requests to lift immunity: authority, offence, outcome | items + committee report | CC BY 4.0 | ✅ `parlament.items` |
+| – written questions and answers | who asked which minister what | item list (J / AB) | CC BY 4.0 | ✅ `parlament.items` |
+| – investigative committees (U-Ausschüsse) | **hearings**: who testified when | item list (KOMM) | CC BY 4.0 | ✅ `parlament.items` |
+| Parlamentskorrespondenz | official press releases (full text); people linked by parliament id where the release links them, otherwise by the entity linker | list 110 + JSON | CC BY 4.0 | ✅ `parlament.press` |
+| Parliament calendar | sessions, committees, hearings, events | list 600 | CC BY 4.0 | ✅ `parlament.calendar` |
+| Rechnungshof – donations | **party donations** (CSV per year/election) | CSV downloads | official works (§7 UrhG) | ✅ `rechnungshof.donations` |
+| Rechnungshof – party annual reports | income/expense totals, **public funding**, donations, sponsoring, ads, **loans**, **holdings**, **affiliated organisations** | XLSX + PDF per party and year | official works | ✅ `rechnungshof.party_reports` |
+| Bundeskanzleramt – government news | summits, trips abroad, bilateral meetings | HTML archive per year (robots.txt only blocks /401) | official works | ✅ `bundesregierung.news` |
+| Bundeskanzleramt – Council of Ministers minutes | every agenda item with the **reporting minister(s)**, file number and subject | HTML per session | official works | ✅ `bundesregierung.ministerrat` |
+| WKStA press releases | official statements on corruption proceedings (stored as documents; legal claims only after review) | HTML | official works | ✅ `justiz.wksta` |
+| Public procurement – data.gv.at "Vergabe-Kerndaten" | **contract awards**: buyer, winner, amount, CPV (eForms) | OGD hub catalogue → per-publisher XML | open data (CC BY 4.0) | ✅ `vergabe.awards` |
+| RTR / KommAustria Medientransparenz | **government advertising** and **media subsidies** per payer → medium | open-data API | free re-use, attribution | ✅ `rtr.medientransparenz` |
+| European Parliament | Austrian MEPs, roles, **declarations of interests** | open-data/website XML | CC BY 4.0 | ✅ `europarl.meps` |
+| European Commission – meetings | meetings of Commissioners/cabinets with **interest representatives** (Austrians and Austrian organisations) | XLSX export | EU re-use policy | ✅ `eu.commission_meetings` |
+| Wikipedia (de/en) | anonymous edits from **parliament/government IP ranges** (`config/ip_ranges.yaml`) | MediaWiki API `usercontribs` | CC BY-SA (edit metadata) | ✅ `wikipedia.institution_edits` |
+| Google Political Ads Transparency | Austrian political ads: advertiser, spend bracket, impressions | public bundle (zip, CSV) | Google's transparency report terms | ✅ `ads.google` |
+| Meta Ad Library API | political ads of configured pages (`config/meta_pages.yaml`) | Graph API, token after identity confirmation | Meta platform terms | ✅ `ads.meta` (opt-in) |
+| RIS (data.bka.gv.at) | laws, court decisions | OGD API | CC BY 4.0 | 🟡 decisions are anonymised; only linkable via a case number cited elsewhere |
+| Lobbying- und Interessenvertretungs-Register (BMJ) | lobbying firms & clients | Lotus Notes web application without stable URLs (deep links return 404) | official work | ⛔ technically not reliably accessible; revisit if an export appears |
+| State parliaments & governments (9 Länder) | Landtag members, votes | varies; several publish OGD | mostly CC BY 4.0 | 🟡 one adapter per state; until then via Wikidata |
+| Municipalities | mayors, councils | fragmented | – | 🟡 via Wikidata |
+| EU Council participant lists | which minister attended which Council | PDF per meeting | EU re-use policy | 🟡 not implemented yet |
+| AMA farm subsidies, EU state-aid register (TAM) | subsidy recipients | search interfaces | – | 🟡 not implemented yet (bulk access and terms still to be checked) |
+| Election results, candidate lists | results per municipality, candidates | BMI / data.gv.at, official gazettes | open data / official | 🟡 not implemented yet |
+| Insolvency notices (Ediktsdatei) | companies tied to politicians | Justiz web application | terms to be checked before automating | 🟡 |
+| Firmenbuch, Grundbuch, beneficial owners (WiEReG) | company roles, property | paid clearing houses; WiEReG public access restricted since the 2022 CJEU ruling | – | ⛔ cost and legal limits |
+| Criminal record (*Strafregister*) | – | not public | – | ⛔ does not exist publicly |
+
+## NGOs & aggregators
+
+| Source | Data | Access | Terms | Status |
+|---|---|---|---|---|
+| Wikidata | positions incl. Landtage and mayors, party membership, careers, birth dates, **social-media handles**, ids | SPARQL | CC0 | ✅ `wikidata.politicians` |
+| GDELT DOC 2.0 | article metadata for politicians' names, attributed to the original outlets | API, ≤ 1 request / 5 s | free | ✅ `gdelt.coverage` (supplement; thin for Austrian outlets) |
+| HowTheyVote.eu | EP roll-call votes | CSV dump | check licence | 🟡 |
+| OpenSanctions | PEP cross-references | bulk JSON | CC BY-NC 4.0 | 🟡 non-commercial only |
+| MeineAbgeordneten.at | curated dossiers | – | sui-generis database right (§76c UrhG) | ⛔ scraping → 🟡 ask for a partnership |
+| Parteispenden.at | party-finance analysis | – | ask | 🟡 partnership |
+
+## Media
+
+The large Austrian outlets do not allow systematic extraction of their articles, and several declare TDM
+reservations. We therefore (`news.feeds`, `config/feeds.yaml`):
+
+* only read the **RSS feeds** and **Google-News sitemaps** they publish for automated discovery,
+* store **headline, teaser (≤ 400 characters) / sitemap keywords and link** only (`store_fulltext=False`,
+  `keep_raw=False` for every media `SourceSpec`), never the article body,
+* for sites with a TDM reservation (e.g. ots.at) keep only headline and link,
+* always send readers to the original article.
+
+| Outlet | Feed / sitemap |
+|---|---|
+| ORF | `rss.orf.at/news.xml` |
+| DER STANDARD | RSS `inland` + news sitemap (robots.txt blocks AI crawlers by name; we only use RSS/sitemap) |
+| Die Presse, KURIER | RSS + news sitemap |
+| profil, Salzburger Nachrichten, Kleine Zeitung | news sitemap listed in robots.txt |
+| FALTER, Krone | RSS |
+| APA-OTS | RSS; TDM reserved → headline + link only. Full-text licence available from APA. |
+
+**Legal caveat:** whether automated name-matching on headlines counts as TDM under §42h UrhG is not
+settled. The conservative implementation keeps the risk low; for more, licence content (APA) or ask outlets.
+
+## Social media
+
+Accounts come from parliament profiles (verified), Wikidata (unverified until confirmed) and the curated
+`config/social_accounts.yaml` (party channels and accounts you checked; `config.social_accounts`).
+
+| Platform | Access | Status |
+|---|---|---|
+| Bluesky | public AppView API (`public.api.bsky.app`), open protocol | ✅ `social.bluesky` |
+| Mastodon / Fediverse | public REST API of the account's instance | ✅ `social.mastodon` |
+| Telegram – public broadcast channels | public web preview `t.me/s/<channel>` (no login, no robots.txt restriction); latest posts | ✅ `social.telegram` |
+| YouTube | Data API v3 (free quota, `POLITRACK_YOUTUBE_API_KEY`) | ✅ `social.youtube` (opt-in) |
+| X (Twitter) | **official API v2 only** (paid read access, `POLITRACK_X_BEARER_TOKEN`). No raw archive or full text kept; posts deleted on X must be removed (deletion sync still open, see roadmap) | ✅ `social.x` (opt-in) |
+| Facebook / Instagram | ToS forbid scraping; Meta Content Library is for researchers only | ⛔ posts · ✅ political ads via the Ad Library API |
+| TikTok | Research API (academic only) | ⛔ |
+
+### Messaging platforms: Telegram, WhatsApp, Signal
+
+Some Austrian politicians and parties run broadcast channels on messengers. What we found and how we treat it:
+
+* **Telegram** – the most used one in Austrian politics, mainly by the FPÖ and the MFG. Reported
+  examples: Herbert Kickl's channel had about 31,700 subscribers in October 2023
+  ([profil](https://www.profil.at)), later about 33,000; the FPÖ channel "Festung Österreich" about
+  19,000–21,000 ([kontrast.at](https://kontrast.at)); FPÖ TV has been on Telegram since 2019
+  ([DER STANDARD](https://www.derstandard.at)). Public channels we could verify with a web preview:
+  `t.me/s/MFGOesterreich` (MFG, ≈ 11,000 subscribers, linked from mfg-oe.at) and `t.me/s/dominikNepp`
+  (Dominik Nepp, FPÖ Wien). Channels that disable the web preview (e.g. Kickl's) cannot be read without the
+  Telegram client/API and an account – we don't do that. Beware of look-alike channels (`herbert_kickl` with
+  2 subscribers is not his).
+  *Policy:* public broadcast channels via the web preview only, low frequency; never private groups or chats.
+* **WhatsApp** – Channels exist (e.g. campaign channels such as Van der Bellen's in the 2022 presidential
+  campaign), but WhatsApp offers no public web view or API for reading channels, and its terms forbid
+  automated collection. ⛔ Not collected. Link a channel in `social_accounts.yaml` only as information.
+* **Signal** – private by design; no public channels or web views exist. ⛔ Nothing to collect.
+
+Context: [Democracy Reporting International](https://democracy-reporting.org) describes Telegram as a way
+for parties to reach supporters outside the content rules of mainstream platforms – one reason to include it.
+
+## Approaches we deliberately do not use
+
+**Scraping full news articles:** even where robots.txt allows crawling, storing or mining article text
+conflicts with publishers' terms and, where declared, their TDM reservation (§42h UrhG). Headlines,
+teasers and sitemap keywords plus a link give us *who/what/when*. Legal ways to get more: licences
+(APA-OTS full text, **APA-DeFacto**), library access (Austrian National Library), agreements with outlets.
+
+**X/Twitter proxies, mirrors or "viewer" services (Nitter, xcancel and similar):** ⛔. They work by
+scraping X against its terms, so using them means relying on a ToS violation – the exact risk to avoid.
+They also break constantly. Legal options: the official API (`social.x`), X's oEmbed for displaying a
+specific post someone links to, the politicians' cross-posts on Bluesky/Mastodon/websites, and Internet
+Archive copies as evidence links for individual deleted posts.
+
+**Paywall bypasses** (archive.ph/12ft-style services, cookie or user-agent tricks, "reader" proxies): ⛔.
+Circumventing technical protection measures can violate §90c UrhG and the publishers' terms. Instead:
+link to the paywalled article, prefer free and public-service sources (ORF, Parlamentskorrespondenz, OTS
+headlines, Dossier, Kobuk, Moment), and get legitimate access for in-depth work. **ANNO** (Austrian
+National Library) offers digitised historical newspapers for free.
+
+**Telegram client API / user accounts, WhatsApp, Signal, private groups:** ⛔ (see above).
+
+## Community submissions
+
+Users can submit a link backing a claim (for example a regional newspaper reporting on a mayor).
+Submissions are moderated and get a low default trust (30) until verified. 🟡