Politrack: track Austrian politicians from public sources
Initial import of Politrack, a self-hosted service that tracks Austrian politicians from federal to municipal level - roles, votes, money flows, statements, meetings, social media and legal proceedings - with every claim linked to its sources. - backend: FastAPI + Postgres/pgvector; adapter framework (fetch -> parse -> load -> enrich) with robots.txt/TDM-aware fetching, a raw-response archive for reparsing, and a claim/evidence model with corroboration - sources: Parliament, Wikidata, EU Parliament/Commission, Council of Ministers, Rechnungshof party reports and donations, procurement, RTR media transparency, political ad libraries, WKStA, news feeds, Wikipedia edits, Bluesky, Mastodon, public Telegram channels, YouTube and X (official API) - web: Next.js frontend in German and English with typed API client - mcp: MCP server for AI agents and RAG - docs: architecture, data model, sources, legal notes, Unraid deployment and contributor guides; Docker Compose setup for Unraid Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
simonandClaude Opus 5.5 committed
commit
583ccba31c
187 files changed
+39890
No files matched your search
@@ -0,0 +1,291 @@
|
||||
# Writing a source adapter
|
||||
|
||||
An **adapter** turns one publisher's data into typed records. It never touches HTTP or the database
|
||||
directly: the framework fetches (politely, legally, archived) and the Loader writes. Your job is two
|
||||
functions, `start()` and `parse()`, and a description of the publisher (`SourceSpec`).
|
||||
|
||||
```
|
||||
start(ctx) -> Request ... entry points (list pages, feeds, API queries)
|
||||
│ Fetcher: robots.txt, throttle, TDM check, conditional GET, retries, raw archive
|
||||
▼
|
||||
parse(ctx, resp) -> Record | Request ... PURE: no DB, no network, no clock-dependent output
|
||||
│ Loader: entity resolution, claim identity, evidence, facets, corroboration status
|
||||
▼
|
||||
PostgreSQL
|
||||
```
|
||||
|
||||
Read [architecture/01-design-from-scratch.md](../architecture/01-design-from-scratch.md) §4 for the
|
||||
reasoning; this page is the how-to.
|
||||
|
||||
## 0. Before writing code: the legal checklist
|
||||
|
||||
Answer these in the pull request (they end up in `SourceSpec.tos_notes`):
|
||||
|
||||
1. **Is there an official API, bulk download or feed?** Use it instead of scraping HTML.
|
||||
2. **robots.txt** – the Fetcher enforces it. Never work around it. If a documented public API is
|
||||
listed in robots.txt only because of generic crawler rules (Wikidata SPARQL, the MediaWiki API), add
|
||||
the API prefix to `robots_exempt` and cite the operator's API policy in `tos_notes`.
|
||||
3. **Terms of service / licence** – may automated access happen? May we keep full text? If not,
|
||||
`store_fulltext=False` (only title, short excerpt and link are kept) and usually `keep_raw=False`.
|
||||
4. **TDM reservation** (§42h UrhG, W3C TDMRep) – detected automatically per origin. If set, no text and no
|
||||
raw archive are kept, whatever the spec says.
|
||||
5. **Personal data** – only data about people in their public role. There is no address field in any
|
||||
record type; don't smuggle one into `data`.
|
||||
6. **No circumvention** – no paywall tricks, no mirrors/proxies of platforms whose ToS forbid scraping
|
||||
(e.g. Nitter for X), no logged-in sessions, no private groups.
|
||||
|
||||
If any answer is "no" or "unclear", document the source in [docs/sources.md](../sources.md) under
|
||||
"Approaches we deliberately do not use" instead.
|
||||
|
||||
## 1. Where things go
|
||||
|
||||
```
|
||||
backend/politrack/sources/
|
||||
my_source.py one module per publisher (or a package for large ones, see parlament/)
|
||||
backend/tests/fixtures/<adapter name>/
|
||||
<case>.request.json captured request (url, kind, meta)
|
||||
<case>.html|json|xml|bin captured response body
|
||||
<case>.expected.json golden output of parse()
|
||||
```
|
||||
|
||||
Modules under `politrack/sources` and `politrack/enrich` are imported automatically; `@register` is all
|
||||
it takes. External packages can add adapters through the `politrack.adapters` entry-point group (see
|
||||
`backend/pyproject.toml`).
|
||||
|
||||
## 2. A complete example
|
||||
|
||||
A (fictitious) state parliament publishes its members as JSON and a page per member:
|
||||
|
||||
```python
|
||||
"""Landtag Musterland: members (JSON API) and their declared side jobs (HTML)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
|
||||
from politrack.core import Adapter, Request, Response, RunContext, SourceSpec, register
|
||||
from politrack.core.records import Claim, Evidence, OrgRef, PersonRecord, PersonRef, Role
|
||||
from politrack.text import excerpt
|
||||
|
||||
BASE = "https://www.landtag-musterland.at"
|
||||
SPEC = SourceSpec(
|
||||
slug="landtag-musterland", # stable; referenced by Evidence.source and user filters
|
||||
name="Landtag Musterland",
|
||||
kind="official", # official | ngo | media | social | aggregator | user | derived
|
||||
homepage=BASE,
|
||||
license="Amtliches Werk (§7 UrhG)",
|
||||
access="api", # api | bulk | rss | scrape | manual | derived
|
||||
trust=90, # default trust 0-100 (users can override)
|
||||
min_interval=2.0, # seconds between requests to this host
|
||||
tos_notes="Open JSON API, no terms restricting reuse; robots.txt allows /api/.",
|
||||
)
|
||||
|
||||
|
||||
@register
|
||||
class MusterlandMembers(Adapter):
|
||||
spec = SPEC
|
||||
name = "musterland.members" # <source>.<dataset>; also the CLI/worker task name
|
||||
description = "Members of the Landtag Musterland with mandates and side jobs"
|
||||
schedule = "15 4 * * *" # cron (UTC); None = manual only
|
||||
queue = "scrape" # scrape | social | bulk
|
||||
creates_persons = True # authoritative member list: may create persons
|
||||
parser_version = 1 # bump when parse() output changes, then `politrack reparse`
|
||||
|
||||
def start(self, ctx: RunContext) -> Iterator[Request]:
|
||||
yield Request(url=f"{BASE}/api/members", params={"period": ctx.params.get("period", "current")},
|
||||
kind="list")
|
||||
|
||||
def parse(self, ctx: RunContext, response: Response) -> Iterator[Any]:
|
||||
if response.request.kind == "list":
|
||||
for m in response.json()["members"]:
|
||||
ref = PersonRef(ids={"landtag-musterland": str(m["id"])}, name=m["name"])
|
||||
yield PersonRecord(ref=ref, full_name=m["name"], photo_url=m.get("photo"))
|
||||
yield Claim(
|
||||
claim_type="role", person=ref,
|
||||
summary_de=f"{m['name']} ist Abgeordnete:r zum Landtag Musterland ({m['party']})",
|
||||
summary_en=f"{m['name']} is a member of the Musterland state parliament ({m['party']})",
|
||||
valid_from=m.get("since"),
|
||||
evidence=[Evidence(url=f"{BASE}/abgeordnete/{m['id']}", doc_type="api_record",
|
||||
title=m["name"])],
|
||||
facet=Role(role="mp", level="state", region="Musterland",
|
||||
body=OrgRef(slug="landtag-musterland", name="Landtag Musterland", kind="parliament"),
|
||||
party=OrgRef(name=m["party"], kind="party", create=False),
|
||||
start=m.get("since")),
|
||||
)
|
||||
# follow-up request; meta travels with it to parse()
|
||||
yield Request(url=f"{BASE}/abgeordnete/{m['id']}", kind="profile",
|
||||
meta={"member_id": str(m["id"]), "name": m["name"]})
|
||||
return
|
||||
|
||||
# kind == "profile": HTML page with a side-job table
|
||||
for row in response.soup.select("table.nebentaetigkeiten tr")[1:]:
|
||||
employer, function = (td.get_text(" ", strip=True) for td in row.find_all("td")[:2])
|
||||
yield Claim(
|
||||
claim_type="side_job",
|
||||
person=PersonRef(ids={"landtag-musterland": response.meta["member_id"]}),
|
||||
counterpart_org=OrgRef(name=employer, kind="company"),
|
||||
key=("musterland", response.meta["member_id"], employer, function), # no facet -> explicit key
|
||||
summary_de=f"{response.meta['name']}: {function} bei {employer}",
|
||||
evidence=[Evidence(url=response.url, doc_type="register_entry", quote=excerpt(row.get_text(" "), 300))],
|
||||
)
|
||||
```
|
||||
|
||||
Run it:
|
||||
|
||||
```bash
|
||||
docker compose exec api politrack sources sync # registers SPEC in the DB
|
||||
docker compose exec api politrack run musterland.members --max-requests 20 -p period=current
|
||||
docker compose exec api politrack status # run stats, errors
|
||||
```
|
||||
|
||||
The periodic task appears automatically for the workers (`schedule`, `queue`).
|
||||
|
||||
## 3. The building blocks
|
||||
|
||||
### Request
|
||||
|
||||
| field | purpose |
|
||||
|---|---|
|
||||
| `url`, `method`, `params`, `json`, `headers` | the HTTP request. Secrets go in `params` named `key`/`token`/`api_key`/`access_token`, or in `headers` – both are redacted from the archive and logs. |
|
||||
| `kind` | dispatch key for `parse()` |
|
||||
| `meta` | context for `parse()` (ids, names, depth); not part of the request identity |
|
||||
| `conditional` | send `If-None-Match`/`If-Modified-Since` from the archive (default on) |
|
||||
| `skip_unchanged` | when body hash and `parser_version` are unchanged, don't reload records – only follow-up requests are collected, so an interrupted run resumes (default on) |
|
||||
|
||||
A request's identity (`fingerprint`) is method + url + params + json. The runner de-duplicates within a run.
|
||||
|
||||
### Response
|
||||
|
||||
`response.text`, `response.json()`, `response.soup` (BeautifulSoup, `html.parser`), `response.meta`
|
||||
(= `request.meta`), `response.url` (final URL, secrets redacted), `response.tdm_reserved`.
|
||||
|
||||
### RunContext
|
||||
|
||||
`ctx.params` (CLI `-p key=value`, worker parameters), `ctx.state` (persistent cursors, see below),
|
||||
`ctx.stats` (a Counter shown in `politrack status` – count what you skip and why), `ctx.log`.
|
||||
`ctx.conn` is available in `start()` only (e.g. to read which social accounts to poll); **`parse()` must
|
||||
not query the database** – that is what keeps it replayable and testable.
|
||||
|
||||
### Records
|
||||
|
||||
All in `politrack/core/records.py`:
|
||||
|
||||
| record | use for |
|
||||
|---|---|
|
||||
| `PersonRecord`, `OrgRecord` | entity data from authoritative sources (names, photo, dates) |
|
||||
| `SocialAccount` | an account of a person/organisation (`verified=True` only if an official page links it) |
|
||||
| `VoteEvent`, `Event` | things with a key that claims point at (votes, sessions, trips) |
|
||||
| `Claim` (+ `facet`) | every fact. Facets: `Role`, `VotePosition`, `Money`, `Participation`, `Statement`, `Legal` |
|
||||
| `Document` | a document without a claim yet (headlines, press releases) – enrichers derive claims later |
|
||||
|
||||
Claim types: `role vote donation money_flow statement appearance meeting travel side_job legal_proceeding
|
||||
lobbying media_mention career parliamentary_question government_decision wiki_edit declaration ownership
|
||||
affiliation party_finance other` (see [data-model.md](../data-model.md)).
|
||||
|
||||
### Referring to people and organisations
|
||||
|
||||
Never look up ids yourself. Describe what you know and let the Loader resolve it:
|
||||
|
||||
```python
|
||||
PersonRef(ids={"parlament": "2339"}) # strong identifier (best)
|
||||
PersonRef(ids={"wikidata": "Q78869"}, name="Alexander Van der Bellen")
|
||||
PersonRef(name="Leonore Gewessler") # exact alias match (titles stripped)
|
||||
PersonRef(hints={"surname_in_body": "nationalrat", "on": date(2024, 3, 1)}, name="Kickl") # roll calls
|
||||
PersonRef(hints={"office": "Bundesministerin für Justiz", "on": date(2025, 6, 17)}) # office holder
|
||||
OrgRef(slug="oevp") # seeded organisations
|
||||
OrgRef(ids={"firmenbuch": "123456a"}, name="Muster GmbH", kind="company") # created if unknown
|
||||
OrgRef(name="ÖVP", kind="party", create=False) # match only, never create
|
||||
```
|
||||
|
||||
Identifier schemes: `parlament`, `wikidata`, `ep`, `uid`, `firmenbuch`, `rtr`, `eu_transparency`,
|
||||
`google_advertiser`, `facebook_page` – add a new scheme by simply using it (document it in data-model.md).
|
||||
Persons are only created by adapters with `creates_persons = True` (authoritative member lists); everyone
|
||||
else's claims about unknown people are skipped and counted as `unresolved_person`.
|
||||
|
||||
Organisation names are matched case-, accent- and legal-form-insensitively (`normalize_org_name`), so
|
||||
"HYPO TIROL BANK AG" and "Hypo Tirol Bank Aktiengesellschaft" become one organisation.
|
||||
|
||||
### Claim identity – facet or `key`
|
||||
|
||||
The Loader computes a claim's identity from *resolved* entities and the facet: a donation is
|
||||
(kind, payer, payee, amount, date, period, reference), a vote position is (vote, person/party), a role is
|
||||
(person, body, role, start), a statement is (person/org, url), a participation is (event, person/org), a
|
||||
media mention is (url, person/org). So the same fact reported by two sources becomes **one claim with two
|
||||
pieces of evidence** – that is how corroboration works.
|
||||
|
||||
Claims **without a facet** (side jobs, careers, declarations, holdings, questions …) need an explicit
|
||||
`key` – the Loader refuses them otherwise. Build it from what identifies the fact in the real world, not
|
||||
from your source's internals, so that another source could produce the same key:
|
||||
`key=("ministerrat", url, file_number, office)`, `key=("side_job", member_id, employer, function)`.
|
||||
A key always overrides the facet-based identity.
|
||||
|
||||
### Summaries
|
||||
|
||||
`summary_de` is required and shown to users; `summary_en` is optional but appreciated. Write neutral,
|
||||
factual one-liners ("X erhielt 2024 eine Spende von Y über 5.000 €"). Use `politrack.text.format_eur_de`
|
||||
for German amounts and `excerpt()` for quotes.
|
||||
|
||||
### Sensitive claims
|
||||
|
||||
Anything about criminal or legal matters: `claim_type="legal_proceeding"`, `sensitive=True`,
|
||||
`review=True` unless the source is an official court/parliament record. The UI then shows the
|
||||
presumption-of-innocence notice and the claim waits for human review.
|
||||
|
||||
## 4. Incremental adapters and cursors
|
||||
|
||||
For large or append-only sources, keep a cursor:
|
||||
|
||||
```python
|
||||
def start(self, ctx):
|
||||
since = ctx.params.get("since") or ctx.state.get("last_date") or "2024-01-01"
|
||||
yield Request(url=..., params={"from": since}, kind="page")
|
||||
|
||||
def parse(self, ctx, response):
|
||||
...
|
||||
ctx.state.set("last_date", newest_date_seen) # the only "write" parse() may do
|
||||
```
|
||||
|
||||
`ctx.state` is persisted per adapter in the `state` table (and kept in memory during tests).
|
||||
|
||||
## 5. Multi-publisher adapters
|
||||
|
||||
Feeds of many outlets, or an aggregator that points at original publishers: define one `SourceSpec` per
|
||||
publisher (register extra specs with `register_spec`) and put `meta={"source": "<slug>"}` on each
|
||||
request – the Fetcher then applies *that* publisher's policy (store_fulltext, throttle) and evidence is
|
||||
attributed to it. See `sources/news.py`.
|
||||
|
||||
## 6. Tests (required)
|
||||
|
||||
1. Capture a fixture from the live source (one request, honouring robots.txt):
|
||||
```bash
|
||||
politrack fixture musterland.members "https://www.landtag-musterland.at/api/members?period=current" list_current --kind list
|
||||
```
|
||||
2. Generate and **review** the golden output:
|
||||
```bash
|
||||
UPDATE_GOLDEN=1 pytest tests/test_golden.py -k musterland
|
||||
git diff tests/fixtures/musterland.members/
|
||||
```
|
||||
3. Unit-test tricky helpers (regexes, date formats) in `tests/test_parsers.py`.
|
||||
|
||||
Keep fixtures small (cut lists to a few entries) and free of personal data beyond public roles.
|
||||
|
||||
## 7. Changing an existing parser
|
||||
|
||||
1. Change `parse()`, bump `parser_version`.
|
||||
2. `UPDATE_GOLDEN=1 pytest` and review the diff.
|
||||
3. After deployment: `politrack reparse <adapter>` replays the raw archive – no refetching. At the end of a
|
||||
complete, error-free reparse, the adapter's claims that the new parser no longer produces are deleted
|
||||
(unless another source backs them), so a fix that changes a claim's identity leaves no stale duplicate.
|
||||
Two adapters sharing one `SourceSpec` should not emit the same claims, or a reparse of one removes the
|
||||
other's until its next run.
|
||||
|
||||
## 8. Checklist for the pull request
|
||||
|
||||
- [ ] `SourceSpec` with honest `kind`, `trust`, `license`, `tos_notes` (legal checklist answered)
|
||||
- [ ] `parse()` is pure; counts skipped rows in `ctx.stats`
|
||||
- [ ] strong identifiers wherever the source has them
|
||||
- [ ] golden fixture + expected output, helper unit tests
|
||||
- [ ] entry in [docs/sources.md](../sources.md)
|
||||
- [ ] new claim types / identifier schemes documented in [data-model.md](../data-model.md) and the
|
||||
frontend (`ClaimDetails.tsx`, messages)
|
||||
@@ -0,0 +1,45 @@
|
||||
# Extending the API (and the MCP server)
|
||||
|
||||
Data flows out through three layers that are kept deliberately thin:
|
||||
|
||||
```
|
||||
politrack/queries/<domain>.py SQL, returns dicts (shared by REST API and MCP server)
|
||||
politrack/api/schemas.py Pydantic response models (the contract → OpenAPI → TypeScript)
|
||||
politrack/api/main.py FastAPI endpoints (parameters, 404s, response_model)
|
||||
politrack/mcp_server.py MCP tools for AI agents (same query functions)
|
||||
```
|
||||
|
||||
## Adding an endpoint, step by step
|
||||
|
||||
1. **Query** – add a function to the matching module in `politrack/queries/` (people, organizations,
|
||||
claims, votes, money, search, admin) and export it from `queries/__init__.py`.
|
||||
* Only show visible claims: use `VISIBLE` (published, not merged) from `queries/common.py`.
|
||||
* Respect the user's source filter: take a `SourceFilter` and use `evidence_json(source_sql)` to
|
||||
aggregate evidence – claims whose evidence is entirely filtered out must disappear.
|
||||
* Always parameterise (`%(name)s`), never format user input into SQL. `LIMIT/OFFSET` on lists.
|
||||
2. **Response model** – add a class to `api/schemas.py` (inherit from `Model`). Field names = column names.
|
||||
3. **Endpoint** – in `api/main.py`, with `response_model=...`. Reuse the dependencies `DB`, `Sources`,
|
||||
`Limit`, `Offset`; comma-separated list parameters go through `csv_list()`.
|
||||
4. **Types for the web app**:
|
||||
```bash
|
||||
cd backend && politrack openapi --ts ../web/src/lib/api-types.ts
|
||||
```
|
||||
Then add a typed function to `web/src/lib/api.ts`. Commit the regenerated `api-types.ts`.
|
||||
5. **MCP tool** (if agents should use it) – a `@mcp.tool()` in `mcp_server.py` with a docstring that tells
|
||||
the agent what the data means and how to cite it.
|
||||
6. **Test** – extend the API smoke test (see [testing.md](testing.md)).
|
||||
|
||||
## Conventions
|
||||
|
||||
* URLs: `/api/v1/<plural noun>[/<slug>[/<sub-resource>]]`; slugs, never internal ids, for people,
|
||||
parties and organisations (vote events use ids – they have no natural slug).
|
||||
* Every claim-like object carries `evidence: [{url, title, source, published_at, quote, …}]` and
|
||||
`source_count`.
|
||||
* Legal matters always include `legal_stage`; the client decides about the presumption-of-innocence
|
||||
notice (`needsInnocenceNotice` in the web app, the `instructions` in the MCP server).
|
||||
* Admin endpoints require `Authorization: Bearer $POLITRACK_ADMIN_TOKEN` and are excluded from the schema.
|
||||
|
||||
## Read-only database access for agents and BI tools
|
||||
|
||||
The role `politrack_reader` (password `POLITRACK_READER_PASSWORD`) can `SELECT` everything. Useful views:
|
||||
`claim_with_evidence`, `person_status`, `person_identifier`. See [../data-model.md](../data-model.md).
|
||||
@@ -0,0 +1,74 @@
|
||||
# Writing an enricher
|
||||
|
||||
An **enricher** derives new facts from data that is already in the database: linking headlines to
|
||||
people, detecting trips and meetings, classifying legal matters, tagging topics, corroborating claims,
|
||||
computing embeddings. Adapters *collect*; enrichers *connect*.
|
||||
|
||||
```python
|
||||
from collections.abc import Iterator
|
||||
|
||||
from politrack.core import Enricher, RunContext, register
|
||||
from politrack.core.records import Claim, Evidence, PersonRef
|
||||
from politrack.core.spec import DERIVED
|
||||
|
||||
|
||||
@register
|
||||
class AmbassadorAppointments(Enricher):
|
||||
spec = DERIVED # derived claims are attributed to "politrack" + the original evidence
|
||||
name = "enrich.ambassadors"
|
||||
description = "Ambassador appointments mentioned in Council of Ministers minutes"
|
||||
schedule = "20 6 * * *" # cron (UTC); runs on the `enrich` queue
|
||||
|
||||
def run(self, ctx: RunContext) -> Iterator[Claim]:
|
||||
rows = ctx.conn.execute(
|
||||
"""SELECT d.id, d.url, d.title, d.published_at, d.full_text FROM document d
|
||||
JOIN source s ON s.id = d.source_id
|
||||
WHERE s.slug = 'bundesregierung' AND d.full_text ILIKE '%%Botschafter%%'
|
||||
AND NOT (d.meta ? 'ambassadors_at')"""
|
||||
).fetchall()
|
||||
for doc in rows:
|
||||
for name in find_appointees(doc["full_text"]): # a pure, unit-tested helper
|
||||
yield Claim(
|
||||
claim_type="career", person=PersonRef(name=name),
|
||||
summary_de=f"{name} als Botschafter:in vorgeschlagen",
|
||||
occurred_at=doc["published_at"],
|
||||
evidence=[Evidence(url=doc["url"], title=doc["title"], source="bundesregierung",
|
||||
quote=..., doc_type="protocol")],
|
||||
)
|
||||
ctx.conn.execute("UPDATE document SET meta = meta || jsonb_build_object('ambassadors_at', now()) "
|
||||
"WHERE id = %s", (doc["id"],))
|
||||
ctx.stats["documents"] += 1
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
* **Yield records, don't write facts yourself.** Records go through the same Loader as adapter output:
|
||||
entity resolution, claim identity, evidence, corroboration status. Direct SQL is fine for *housekeeping*
|
||||
(marking documents as processed, maintenance like closing roles at death – see `enrich/maintenance.py`).
|
||||
* **Cite the original.** Evidence of a derived claim points at the original document and names its
|
||||
`source` slug, so the user's source filter and the corroboration logic work on the real publisher.
|
||||
* **Be incremental.** Remember what you processed (a marker in `document.meta`, or `ctx.state`), so a run
|
||||
costs proportional to new data.
|
||||
* **Be conservative with people.** Enrichers never create persons (`creates_persons = False`); unknown
|
||||
names are skipped and counted (`unresolved_person`). Prefer precision over recall – a missed link is
|
||||
better than a wrong one.
|
||||
* **Sensitive output needs review.** Anything about legal matters: `sensitive=True, review=True`.
|
||||
* **Put logic in pure helpers** (regexes, classifiers) and unit-test them in `tests/test_extraction.py`.
|
||||
|
||||
The runner commits every 200 records, so a crash loses little work.
|
||||
|
||||
## Existing enrichers
|
||||
|
||||
| name | what it does |
|
||||
|---|---|
|
||||
| `enrich.linker` | finds politicians (aliases, "Bundeskanzler Stocker" office titles) in headlines/teasers → `media_mention` claims |
|
||||
| `enrich.events` | trips, summits, bilateral meetings in official news → `Event` + participation, `travel`/`meeting` claims |
|
||||
| `enrich.legal` | investigation/indictment/verdict vocabulary in headlines → `legal_proceeding` claims (review queue) |
|
||||
| `enrich.topics` | keyword topics (mirrors Parliament's subject list) → `topic_assignment` |
|
||||
| `enrich.roles` | gives open-ended offices the holder's party; closes roles at death |
|
||||
| `enrich.rollcall` | faction majority positions derived from individual roll-call votes |
|
||||
| `enrich.corroborate` | merges near-duplicate claims, recomputes reported/corroborated/disputed |
|
||||
| `enrich.embed` | chunks + embeddings for semantic search/RAG (needs Ollama) |
|
||||
| `config.social_accounts` | imports `config/social_accounts.yaml` (curated accounts, party channels) |
|
||||
|
||||
Run them all with `politrack enrich`, or one with `politrack run enrich.linker`.
|
||||
@@ -0,0 +1,81 @@
|
||||
# Extending the web app
|
||||
|
||||
Next.js 15 (App Router, React Server Components) with next-intl (German default, English). Pages fetch
|
||||
from the API on the server; there is almost no client-side JavaScript.
|
||||
|
||||
```
|
||||
web/
|
||||
messages/de.json, en.json all UI text (keys must exist in both)
|
||||
src/app/[locale]/<route>/page.tsx pages (German route names: /personen, /parteien, /geld …)
|
||||
src/components/ shared server components
|
||||
src/lib/api.ts typed API client (server-side only; reads the filter cookie)
|
||||
src/lib/api-types.ts GENERATED from the backend's response models – do not edit
|
||||
src/lib/format.ts dates, money, localized names
|
||||
src/lib/prefs.ts the user's source/topic filter (cookie)
|
||||
```
|
||||
|
||||
## Types come from the backend
|
||||
|
||||
After any change to `backend/politrack/api/schemas.py`:
|
||||
|
||||
```bash
|
||||
cd backend && politrack openapi --ts ../web/src/lib/api-types.ts
|
||||
cd ../web && npm run typecheck
|
||||
```
|
||||
|
||||
`api.ts` re-exports the generated types (`Claim`, `MoneyFlow`, …) and adds one function per endpoint.
|
||||
`api.claims()`, `api.money()`, `api.social()` … automatically apply the visitor's source filter
|
||||
(`withPrefs`), so every list respects "which sources do I trust".
|
||||
|
||||
## Adding a page
|
||||
|
||||
1. Create `src/app/[locale]/<route>/page.tsx` – an async server component:
|
||||
```tsx
|
||||
export default async function Page({params, searchParams}: Props) {
|
||||
const {locale} = await params;
|
||||
setRequestLocale(locale);
|
||||
const t = await getTranslations('MyPage');
|
||||
const rows = await api.something({...});
|
||||
return <>…</>;
|
||||
}
|
||||
```
|
||||
Filters are plain `<form method="get">` – shareable URLs, no client state.
|
||||
2. Add the texts to **both** `messages/de.json` and `messages/en.json`.
|
||||
3. Link it from the navigation (`src/app/[locale]/layout.tsx`) or the home page's "Entdecken" cards.
|
||||
4. Page modules may only export Next.js page fields (`default`, `generateMetadata`, …); keep helpers and
|
||||
constants module-local or in `src/components`.
|
||||
|
||||
## Showing a new claim type
|
||||
|
||||
Claims render through `ClaimList`; type-specific details come from the registry in
|
||||
`src/components/ClaimDetails.tsx`:
|
||||
|
||||
```tsx
|
||||
const ambassador: Renderer = (claim, {t}) => (
|
||||
<p className="claim-detail small">{t('country')}: {String(claim.data.country ?? '')}</p>
|
||||
);
|
||||
|
||||
const DETAILS: Partial<Record<ClaimType, Renderer[]>> = {
|
||||
…,
|
||||
career: [counterpart, ambassador],
|
||||
};
|
||||
```
|
||||
|
||||
Also add the label under `Claims.type.<claim_type>` in both message files, and (if it should appear) the
|
||||
type to the person page's `TABS`.
|
||||
|
||||
## Non-negotiables in the UI
|
||||
|
||||
* **Every fact shows its sources** (the "Quellen" disclosure under each claim, `[source]` links in
|
||||
tables). Don't render a claim without its evidence.
|
||||
* **Presumption of innocence:** `needsInnocenceNotice()` in `ClaimList.tsx` decides; never remove the
|
||||
notice for anything but an undisputed final conviction.
|
||||
* **Connections are not accusations:** pages that show networks or money of connected organisations
|
||||
carry the note (`Network.note`, `Organizations.connectionNote`).
|
||||
* Unverified social accounts are marked (Wikidata-only handles).
|
||||
|
||||
## Styling
|
||||
|
||||
One global stylesheet (`src/app/globals.css`) with CSS variables for light/dark mode. Reuse the existing
|
||||
classes (`claims`, `claim-detail`, `chip`, `tag`, `notice`, `grid-2`, `compact`, `num`, `muted`, `small`)
|
||||
before adding new ones.
|
||||
@@ -0,0 +1,50 @@
|
||||
# Testing
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
pytest # everything below except the DB smoke test
|
||||
pytest tests/test_golden.py -k parlament # one adapter's golden tests
|
||||
UPDATE_GOLDEN=1 pytest tests/test_golden.py # regenerate expected outputs (review the diff!)
|
||||
```
|
||||
|
||||
| suite | what it covers | needs |
|
||||
|---|---|---|
|
||||
| `tests/test_core.py` | framework invariants: request identity, secret redaction, registry consistency, record serialisation, OpenAPI→TS generator, shared helpers | nothing |
|
||||
| `tests/test_parsers.py` | parsing helpers (vote texts, CSV layouts of the Rechnungshof, dates, normalisation) | nothing |
|
||||
| `tests/test_extraction.py` | roll-call lists, immunity reports, event/legal/office-title detection | nothing |
|
||||
| `tests/test_golden.py` | every captured fixture replayed through `adapter.parse()` and compared with `<case>.expected.json` | nothing |
|
||||
|
||||
Parsers are pure, so none of these needs a database or network.
|
||||
|
||||
## Golden fixtures
|
||||
|
||||
```
|
||||
tests/fixtures/<adapter name>/<case>.request.json {url, kind, meta, content_type, body_file}
|
||||
tests/fixtures/<adapter name>/<case>.<ext> the captured body
|
||||
tests/fixtures/<adapter name>/<case>.expected.json parse() output (records as JSON)
|
||||
```
|
||||
|
||||
Capture with `politrack fixture <adapter> <url> <case> --kind <kind> --meta '{"…": …}'`. A fixture
|
||||
without `.expected.json` creates it on the first run and is skipped until you have reviewed it.
|
||||
|
||||
When a source changes its format: capture a new case next to the old one (keep the old one if the
|
||||
publisher still serves old documents in that format), fix the parser, bump `parser_version`.
|
||||
|
||||
## Against a real database
|
||||
|
||||
The quickest end-to-end check uses the compose stack:
|
||||
|
||||
```bash
|
||||
docker compose up -d db && docker compose run --rm migrate
|
||||
docker compose run --rm api politrack run parlament.persons --max-requests 20 -p periods='["XXVIII"]'
|
||||
docker compose run --rm api politrack enrich
|
||||
docker compose run --rm api politrack status
|
||||
curl -s localhost:8000/api/v1/stats
|
||||
```
|
||||
|
||||
`--max-requests` keeps test runs polite. Each run is recorded in the `run` table (stats, errors) and
|
||||
shown by `politrack status` and `/api/v1/admin/runs`.
|
||||
|
||||
## Frontend
|
||||
|
||||
`npm run typecheck` (strict TypeScript against the generated API types) and `npm run build`.
|
||||
Reference in new issue
Block a user