Politrack: track Austrian politicians from public sources
Initial import of Politrack, a self-hosted service that tracks Austrian politicians from federal to municipal level - roles, votes, money flows, statements, meetings, social media and legal proceedings - with every claim linked to its sources. - backend: FastAPI + Postgres/pgvector; adapter framework (fetch -> parse -> load -> enrich) with robots.txt/TDM-aware fetching, a raw-response archive for reparsing, and a claim/evidence model with corroboration - sources: Parliament, Wikidata, EU Parliament/Commission, Council of Ministers, Rechnungshof party reports and donations, procurement, RTR media transparency, political ad libraries, WKStA, news feeds, Wikipedia edits, Bluesky, Mastodon, public Telegram channels, YouTube and X (official API) - web: Next.js frontend in German and English with typed API client - mcp: MCP server for AI agents and RAG - docs: architecture, data model, sources, legal notes, Unraid deployment and contributor guides; Docker Compose setup for Unraid Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
commit
583ccba31c
187 files changed
+39890
No files matched your search
@@ -0,0 +1,325 @@
|
||||
# Politrack – clean-room design
|
||||
|
||||
> Written as if no code existed. Inputs were only the requirements: track what Austrian
|
||||
> politicians (federal → municipal) do, from many heterogeneous public sources, legally, with
|
||||
> every fact backed by links; user-selectable trust & topics; person/party overviews; calendar
|
||||
> and timeline; multilingual (de default, en); agent/RAG access; runs on an Unraid homelab in
|
||||
> Docker; scrapers deployable separately; easy for contributors to extend.
|
||||
>
|
||||
> [02-comparison-and-decisions.md](02-comparison-and-decisions.md) compares this design with
|
||||
> the first implementation and records which decisions won.
|
||||
|
||||
## 1. Quality goals (in priority order)
|
||||
|
||||
1. **Trustworthiness** – no fact without provenance; never present something as more certain
|
||||
than its sources; sensitive facts (criminal matters) are gated. Precision beats recall.
|
||||
2. **Legal compliance by construction** – the framework, not the individual scraper author,
|
||||
enforces robots.txt, TDM reservations, rate limits, storage limits and data minimisation.
|
||||
3. **Reproducibility** – any derived row can be explained ("which source, which response,
|
||||
which parser version") and regenerated from archived raw data after a parser fix, without
|
||||
hitting the source again.
|
||||
4. **Extensibility** – adding a source must touch exactly one module (+ a fixture-based test
|
||||
and a docs line). No edits to SQL seeds, schedulers or registries elsewhere.
|
||||
5. **Operability on a homelab** – few moving parts (one database, one image), self-healing
|
||||
schedules, everything inspectable with SQL.
|
||||
|
||||
## 2. System context
|
||||
|
||||
```
|
||||
┌───────────── public sources (APIs, open data, feeds, sitemaps, archives) ─────────────┐
|
||||
▼ │
|
||||
┌────────────────┐ Request ┌──────────┐ Response ┌──────────┐ Records ┌────────┐ │
|
||||
│ Adapter.start()│──────────▶ │ Fetcher │─────────▶ │ Adapter. │───────▶ │ Loader │──▶ Postgres│
|
||||
│ (per source) │◀────────── │ (policy, │ archive │ parse() │ FollowUp│ │ │
|
||||
└────────────────┘ FollowUp │ archive) │──▶ Raw └──────────┘ └────────┘ │
|
||||
└──────────┘ Archive (blobs + fetch_log) ▲ │
|
||||
│ Records │
|
||||
Enrichers (linking, events, legal, topics, ─────┘ │
|
||||
corroboration, embeddings) read Postgres │
|
||||
│
|
||||
Serving: FastAPI (typed JSON, OpenAPI) ─▶ Next.js web (de/en) MCP server ─▶ AI agents │
|
||||
read-only SQL role ─▶ notebooks/BI/agents pgvector chunks ─▶ RAG │
|
||||
```
|
||||
|
||||
Four strictly separated stages:
|
||||
|
||||
| Stage | Input → output | Side effects | Determinism |
|
||||
|---|---|---|---|
|
||||
| **Fetch** | `Request` → `Response` | network; raw archive write | no |
|
||||
| **Parse** | `Response` → `Record`s and follow-up `Request`s | none (pure) | yes, versioned |
|
||||
| **Load** | `Record` → canonical rows | DB writes (single code path) | yes, idempotent |
|
||||
| **Enrich** | canonical rows → `Record`s | via Loader only | yes, idempotent |
|
||||
|
||||
Because parse is pure and responses are archived, `politrack reparse <adapter>` rebuilds data
|
||||
after a parser fix without re-fetching. Because load is the only writer, invariants (no claim
|
||||
without evidence, data minimisation, identity rules) live in one place.
|
||||
|
||||
## 3. Data model
|
||||
|
||||
### 3.1 Principles
|
||||
|
||||
* **Claim/evidence**: every user-visible fact is a `claim` with ≥1 `claim_evidence` row pointing
|
||||
to a `document` (a URL) published by a `source`. Claims from different sources about the same
|
||||
fact collapse into one claim with several evidence rows ("corroborated").
|
||||
* **Typed facets**: claims of some types carry a structured facet row (role, vote position,
|
||||
money flow, event participation, statement, legal proceeding). Facets make analytics and
|
||||
correlations possible; the claim makes provenance and display uniform.
|
||||
* **Identifiers instead of ID columns**: entities get any number of external identifiers in one
|
||||
table (`scheme`, `value`), e.g. `parlament:2339`, `wikidata:Q123`, `ep:197479`,
|
||||
`bluesky:did:plc:…`, `telegram:HKickl`, `uid:ATU12345678`, `rtr:7846`. New sources add schemes,
|
||||
not columns.
|
||||
* **Bitemporal enough**: claims have `occurred_at` (point events) or `valid_from/valid_to`
|
||||
(states such as mandates); documents have `published_at`; everything has `created_at/updated_at`.
|
||||
|
||||
### 3.2 Tables (abridged; full DDL in `backend/migrations`)
|
||||
|
||||
```
|
||||
source(slug PK-ish, name, kind, homepage, license, access, trust, store_fulltext, keep_raw, tos_notes, …)
|
||||
fetch(id, adapter, source_id, request_fp, url, method, status, sha256, size, content_type,
|
||||
etag, last_modified, fetched_at, request jsonb, parsed_sha256, parser_version) -- raw archive index
|
||||
document(id, source_id, url UNIQUE, title, doc_type, lang, published_at, excerpt, full_text,
|
||||
fetch_id, story_id, meta jsonb)
|
||||
person(id, slug, full_name, given_name, family_name, prefixes…, birth_date, death_date, gender, photo_url)
|
||||
organization(id, slug, name, short_name, kind, parent_id, color, …)
|
||||
identifier(entity_type, entity_id, scheme, value, source_slug, verified) UNIQUE(entity_type, scheme, value)
|
||||
alias(entity_type, entity_id, alias, alias_norm)
|
||||
social_account(id, person_id|organization_id, platform, handle, url, verified, source_slug)
|
||||
claim(id, claim_type, fingerprint UNIQUE, person_id, organization_id, counterpart_person_id,
|
||||
counterpart_org_id, occurred_at, valid_from, valid_to, summary_de, summary_en, data jsonb,
|
||||
status, sensitive, review_status, merged_into, created_by, …)
|
||||
claim_evidence(claim_id, document_id, stance, quote, locator, extracted_by, confidence)
|
||||
role / vote_event / vote_position / money_flow / event / event_participant / statement /
|
||||
legal_proceeding -- facets
|
||||
topic, topic_assignment, story, chunk(text, tsv, embedding vector(1024))
|
||||
run(id, adapter, started_at, finished_at, status, stats, error), state(adapter, key, value)
|
||||
```
|
||||
|
||||
### 3.3 Claim identity
|
||||
|
||||
The *Loader* (not the scraper) computes a claim's fingerprint from **resolved entity ids** and
|
||||
the semantically identifying fields of its type:
|
||||
|
||||
| claim type | identity |
|
||||
|---|---|
|
||||
| role | person, body, canonical role, start date |
|
||||
| vote | vote event key, voter (person or party) |
|
||||
| donation / money_flow | kind, payer, payee, amount, date, external reference |
|
||||
| statement | person, URL |
|
||||
| media_mention | URL, subject |
|
||||
| meeting / travel | person, anchor (counterpart surname or country), ISO week |
|
||||
| legal_proceeding, others | explicit key from the adapter |
|
||||
|
||||
Using resolved ids means a Wikidata mandate and a Parliament mandate of the same person, body
|
||||
and start date *are the same claim* by construction. A fuzzy **corroboration** pass handles
|
||||
near-duplicates (dates off by days, wording differences) by merging claims (`merged_into`).
|
||||
|
||||
### 3.4 Entity resolution
|
||||
|
||||
Records refer to entities through `PersonRef` / `OrgRef`:
|
||||
|
||||
```python
|
||||
PersonRef(ids={"parlament": "2339"}, name="Selma Yildirim") # strong id
|
||||
PersonRef(name="Auer", hints={"surname_in_body": "nationalrat", "on": date(2025, 9, 24)})
|
||||
OrgRef(name="Hypo Tirol Bank AG", kind="company", ids={"uid": "ATU…"})
|
||||
```
|
||||
|
||||
Resolution order: (1) any strong identifier → entity; (2) strategy named in hints
|
||||
(e.g. *surname among members of body on date*); (3) exact normalised alias; (4) for
|
||||
organisations: normalised legal name (legal-form suffixes stripped); (5) create — persons are
|
||||
only created by "authoritative" adapters (`creates_persons=True`), other adapters leave
|
||||
unresolvable person refs unlinked and count them. Ambiguity never guesses.
|
||||
|
||||
## 4. The adapter framework (the nitty-gritty)
|
||||
|
||||
### 4.1 SourceSpec – single source of truth for a publisher
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class SourceSpec:
|
||||
slug: str # "parlament"
|
||||
name: str
|
||||
kind: Literal["official", "ngo", "media", "social", "aggregator", "user"]
|
||||
homepage: str
|
||||
license: str
|
||||
access: Literal["api", "bulk", "rss", "scrape", "manual"]
|
||||
trust: int # default 0–100, editorial
|
||||
store_fulltext: bool # may we keep article/body text?
|
||||
keep_raw: bool = True # may we archive raw responses? (False for TDM-reserved)
|
||||
min_interval: float = 2.0 # seconds between requests to one host
|
||||
robots_exempt: tuple[str, ...] = () # URL prefixes of documented APIs; justify in tos_notes
|
||||
tos_notes: str = ""
|
||||
```
|
||||
|
||||
Specs live next to their adapter. `politrack sources sync` (run by `migrate`) upserts them into
|
||||
the `source` table, so SQL never needs editing to add a source.
|
||||
|
||||
### 4.2 Adapter
|
||||
|
||||
```python
|
||||
class Adapter:
|
||||
spec: ClassVar[SourceSpec]
|
||||
name: ClassVar[str] # "parlament.items" (a source may have several adapters)
|
||||
schedule: ClassVar[str | None] # cron (UTC) or None = manual only
|
||||
queue: ClassVar[str] = "scrape"
|
||||
parser_version: ClassVar[int] = 1 # bump when parse() output changes → reparse
|
||||
|
||||
def start(self, ctx: RunContext) -> Iterable[Request]: ...
|
||||
def parse(self, ctx: RunContext, response: Response) -> Iterable[Record | Request]: ...
|
||||
```
|
||||
|
||||
* `start()` yields the entry requests (list endpoints, index pages, feeds). It may read
|
||||
`ctx.state` (cursors) and `ctx.since`.
|
||||
* `parse()` is **pure**: no DB access, no network. It yields records and follow-up requests
|
||||
(detail pages). Dispatch on `response.request.kind` (`"list"`, `"detail"`, …).
|
||||
* Anything that needs the database (e.g. "who sat in the Nationalrat on that day") is expressed
|
||||
as resolution hints on refs and done by the Loader.
|
||||
|
||||
### 4.3 Request / Response
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class Request:
|
||||
url: str
|
||||
kind: str = "default" # parse() dispatch key
|
||||
method: str = "GET"
|
||||
params: dict | None = None
|
||||
json: Any = None
|
||||
headers: dict | None = None
|
||||
meta: dict = field(default_factory=dict) # passed through to parse (e.g. {"gp": "XXVIII"})
|
||||
conditional: bool = True # send If-None-Match / If-Modified-Since
|
||||
skip_unchanged: bool = True # don't reload records if body hash and parser_version unchanged
|
||||
|
||||
@dataclass
|
||||
class Response:
|
||||
request: Request
|
||||
status: int
|
||||
url: str
|
||||
headers: Mapping[str, str]
|
||||
body: bytes
|
||||
fetched_at: datetime
|
||||
sha256: str
|
||||
fetch_id: int | None # row in the raw archive index
|
||||
def text(self) -> str; def json(self) -> Any; def soup(self) -> BeautifulSoup
|
||||
```
|
||||
|
||||
### 4.4 Fetcher (policy enforcement)
|
||||
|
||||
For every request the fetcher:
|
||||
1. checks robots.txt (cached per origin) unless the URL starts with a `robots_exempt` prefix;
|
||||
2. throttles per host (`max(spec.min_interval, robots crawl-delay)`);
|
||||
3. sends an honest User-Agent with contact address, and conditional headers from the last fetch;
|
||||
4. retries 429/5xx with exponential back-off, honouring `Retry-After`;
|
||||
5. detects TDM reservations (`tdm-reservation` header, `/.well-known/tdmrep.json`) and records
|
||||
them on the response (`response.tdm_reserved`) – the Loader then drops full text;
|
||||
6. writes the body to the raw archive (`sha256`-addressed, gzip) **iff** `spec.keep_raw` and no
|
||||
TDM reservation, and logs a `fetch_log` row in every case.
|
||||
|
||||
### 4.5 Records (typed, validated)
|
||||
|
||||
```python
|
||||
Evidence(url, title=None, doc_type="web_page", published_at=None, excerpt=None, full_text=None,
|
||||
quote=None, locator="", stance="supports", confidence=1.0, source=None)
|
||||
PersonRecord(ref, full_name, given_name=None, family_name=None, …, aliases=(), social=())
|
||||
OrgRecord(ref, name, short_name=None, kind="other", parent=None, color=None)
|
||||
Claim(claim_type, summary_de, evidence, person=None, org=None, counterpart_person=None,
|
||||
counterpart_org=None, occurred_at=None, valid_from=None, valid_to=None, summary_en=None,
|
||||
data={}, topics=(), sensitive=False, review=False, key=None, facet=None)
|
||||
# facets
|
||||
Role(role, level, body=None, party=None, region=None, start=None, end=None)
|
||||
VoteEvent(key, chamber, title, voted_on, result, stage=None, item_ref=None, roll_call=False, url=None, …)
|
||||
VotePosition(event_key, position) # voter = claim.person or claim.org
|
||||
Money(kind, amount, currency="EUR", payer=None, payee=None, payer_name=None, payee_name=None,
|
||||
date=None, reference=None, category=None, period=None)
|
||||
Event(key, kind, title, starts_at, ends_at=None, location=None, country=None, url=None, body=None)
|
||||
Participation(event_key, role="participant")
|
||||
Statement(context, url, said_at=None, medium=None, title=None, excerpt=None, platform=None)
|
||||
Legal(stage, offense=None, authority=None, case_ref=None, stage_date=None)
|
||||
Document(evidence) # a document without a claim (e.g. a headline);
|
||||
# enrichers create claims from it later
|
||||
```
|
||||
|
||||
Data minimisation is enforced by *schema*: e.g. `Money` has no address field, so a parser
|
||||
cannot leak a donor's street address into the database even by accident.
|
||||
|
||||
### 4.6 Runner
|
||||
|
||||
```python
|
||||
def run(adapter, mode="fetch", since=None):
|
||||
ctx = RunContext(adapter, since, state=StateStore(adapter.name), stats=Counter())
|
||||
queue = deque(adapter.start(ctx)); seen = set()
|
||||
while queue:
|
||||
req = queue.popleft()
|
||||
if fingerprint(req) in seen: continue; seen.add(fingerprint(req))
|
||||
resp = fetcher.fetch(req) if mode == "fetch" else archive.latest(req)
|
||||
if resp is None: continue
|
||||
if req.skip_unchanged and archive.already_parsed(resp, adapter.parser_version):
|
||||
# records already loaded; still follow links so an interrupted run's children get fetched
|
||||
queue.extend(r for r in adapter.parse(ctx, resp) if isinstance(r, Request)); continue
|
||||
with db.transaction(): # one transaction per response
|
||||
for item in adapter.parse(ctx, resp):
|
||||
queue.append(item) if isinstance(item, Request) else loader.load(item, ctx)
|
||||
archive.mark_parsed(resp, adapter.parser_version)
|
||||
record run row with stats/errors
|
||||
```
|
||||
|
||||
Per-response transactions mean a crash loses one page, never the whole run; errors in one
|
||||
response are recorded and the run continues (error budget → run status `partial`).
|
||||
|
||||
### 4.7 Scheduling & deployment
|
||||
|
||||
* Procrastinate (Postgres-backed queue): the worker builds one periodic task per adapter from
|
||||
`adapter.schedule`/`adapter.queue` – no hand-written task list.
|
||||
* Queues `scrape`, `social`, `bulk`, `enrich` let heavy bulk jobs (procurement, ad archives) run
|
||||
on their own worker (even another machine) without blocking light ones.
|
||||
* One Docker image for api/worker/mcp/migrate; third-party adapter packages register via the
|
||||
Python entry-point group `politrack.adapters`, so a separate deployment can add sources
|
||||
without forking.
|
||||
|
||||
### 4.8 Enrichers
|
||||
|
||||
Same Loader, different input: an `Enricher` has `run(ctx) -> Iterable[Record]` over database
|
||||
rows (documents not yet linked, claims not yet tagged…). Built-ins: mention linker,
|
||||
event extractor (trips/meetings), legal-matter detector (→ review queue), topic tagger,
|
||||
corroborator (merge near-duplicates, cluster news stories), say-vs-vote matcher, chunker +
|
||||
embedder.
|
||||
|
||||
## 5. Serving
|
||||
|
||||
* **API**: FastAPI with Pydantic response models (OpenAPI is the contract). Standard filters
|
||||
everywhere: `person`, `party`, `types`, `topics`, `since`, `until`, `exclude_sources`,
|
||||
`min_trust`, `limit/offset`.
|
||||
* **Web**: Next.js App Router + next-intl (de at `/`, en at `/en`); TypeScript types generated
|
||||
from OpenAPI (`politrack openapi --ts`); preferences in a cookie so server components can
|
||||
filter; a **renderer registry** maps claim types to components (adding a claim type = one
|
||||
component).
|
||||
* **Agents**: MCP tools mirroring the API; read-only SQL role; `COMMENT ON` everything;
|
||||
`claim_with_evidence` view; `chunk` table for RAG with citations.
|
||||
|
||||
## 6. Testing strategy
|
||||
|
||||
| Level | What | How |
|
||||
|---|---|---|
|
||||
| Parser golden tests | each adapter's `parse()` | real responses saved as fixtures (`tests/fixtures/<adapter>/…`), expected records as JSON next to them; `UPDATE_GOLDEN=1` regenerates |
|
||||
| Unit tests | text normalisation, resolvers, identity | plain pytest |
|
||||
| Loader/integration | load records into an ephemeral Postgres, assert rows | `pytest -m integration` with `POLITRACK_TEST_DATABASE_URL` |
|
||||
| API smoke | every endpoint against a seeded DB | FastAPI TestClient |
|
||||
| Frontend | type check + build | `npm run typecheck && npm run build` |
|
||||
|
||||
## 7. Legal & ethical guard-rails in code
|
||||
|
||||
* Fetcher enforces robots/TDM/rate limits; `keep_raw`/`store_fulltext` per source.
|
||||
* Record schemas exclude data we must not keep (addresses, private contact data).
|
||||
* `sensitive=True` claims render the presumption-of-innocence notice unless stage is an
|
||||
undisputed final conviction; auto-detected legal matters require `review`.
|
||||
* Messengers: only *public broadcast channels* (e.g. public Telegram channels' web preview);
|
||||
never private groups, never platforms whose terms forbid automated access (WhatsApp, X without
|
||||
API, Facebook/Instagram).
|
||||
|
||||
## 8. Operations
|
||||
|
||||
* Services: `db` (Postgres 16 + pgvector), `migrate` (one-shot), `api`, `web`, `mcp`,
|
||||
`worker-scrape`, `worker-bulk`, `worker-enrich`, optional `ollama`.
|
||||
* Volumes: Postgres data, raw archive (`/data/raw`), Ollama models.
|
||||
* Observability: `run` table, `politrack status`, `/api/v1/admin/runs` (token), structured logs.
|
||||
* Backups: `pg_dump` + raw archive rsync (raw archive is optional to back up – it is a cache
|
||||
that can be re-fetched, but it is also the audit trail).
|
||||
@@ -0,0 +1,92 @@
|
||||
# Clean-room design vs. first implementation – decisions
|
||||
|
||||
The first implementation (v1) grew iteratively. The clean-room design
|
||||
([01-design-from-scratch.md](01-design-from-scratch.md)) was written without looking at it.
|
||||
This document compares them area by area and records the decision taken for v2 (the current
|
||||
code). Format: lightweight ADRs.
|
||||
|
||||
## Summary
|
||||
|
||||
| # | Area | v1 (first implementation) | Clean-room design | **Decision (v2)** |
|
||||
|---|---|---|---|---|
|
||||
| 1 | Pipeline shape | each scraper fetches, parses and writes in one method | fetch → archive → pure parse → loader | **Clean-room**. Bugs found during v1 testing (vote parser, roll calls, offence regex) required resetting state and re-fetching; with an archive they are a `reparse`. |
|
||||
| 2 | Writing to the DB | scrapers call `store.record_claim(...)` with hand-made fingerprint keys | typed records; Loader computes identity from resolved entity ids | **Clean-room**, keeping v1's battle-tested upsert SQL inside the Loader. |
|
||||
| 3 | Source definition | SQL seed + scraper class + worker schedule + YAML, four places | `SourceSpec` in code, synced to DB; schedule on the adapter | **Clean-room**. One module per source. |
|
||||
| 4 | External ids | columns `parlament_pad`, `wikidata_qid` | `identifier(scheme, value)` table | **Clean-room**. Needed anyway for EP, Bluesky DIDs, Telegram, company UIDs, RTR ids. |
|
||||
| 5 | Entity references | scrapers look persons up themselves (SQL in scrapers) | `PersonRef`/`OrgRef` with ids/name/hints, resolved by the Loader | **Clean-room**. Makes parsers pure and testable. |
|
||||
| 6 | Change detection | per-scraper ad-hoc signatures in `scrape_state` | generic: body hash + parser version per request fingerprint; conditional GET | **Clean-room**; v1's cursor state store kept for real cursors (dates, ids). |
|
||||
| 7 | Policy enforcement | `PoliteClient` (robots, TDM, throttle, retries) + per-call `documented_api=True` | same, but exemptions declared on `SourceSpec.robots_exempt` | **Merge**: v1 client code + declarative exemptions. |
|
||||
| 8 | Corroboration | post-hoc merge engine (roles, stories, events) | identity by construction + fuzzy merge pass | **Both**: identity by construction removes most duplicates; v1's merge engine handles near-duplicates. |
|
||||
| 9 | Activity status | `person_status` view with verified flag and term windows | not specified in detail | **v1** (proven on real data: Van der Bellen/Mikl-Leitner cases). |
|
||||
| 10 | Legal matters | review queue, innocence-notice rule, official immunity data | same principles | **v1** implementation, moved behind the Loader (`review=True`). |
|
||||
| 11 | Scheduling | Procrastinate, tasks written by hand | Procrastinate, tasks generated from adapters; extra `bulk`/`social` queues | **Clean-room**. |
|
||||
| 12 | Enrichment | functions writing claims directly | enrichers emitting records to the Loader | **Clean-room**. |
|
||||
| 13 | API contract | handwritten dict responses; hand-maintained TS types | Pydantic response models, generated TS types | **Clean-room**, generated with a small built-in generator (`politrack openapi --ts`) so it works without npm. |
|
||||
| 14 | Frontend claim rendering | one generic `ClaimList` with `if` branches | renderer registry per claim type | **Clean-room** (generic fallback + specific renderers). |
|
||||
| 15 | Tests | inline-string unit tests | golden tests from real fixtures + unit + integration + API smoke | **Clean-room**, keeping v1's unit tests. |
|
||||
| 16 | Data minimisation | sanitiser function in the donations scraper | enforced by record schemas | **Clean-room** + keep the sanitiser (defence in depth). |
|
||||
| 17 | Storage of raw data | none | content-addressed raw archive, per-source `keep_raw` | **Clean-room**. |
|
||||
| 18 | Plugins | none | entry points `politrack.adapters` | **Clean-room** (cheap to add, enables separate deployments). |
|
||||
| 19 | Stack choices (Postgres+pgvector, FastAPI, Next.js+next-intl, MCP, Docker Compose) | ✓ | ✓ | **Same in both** – confirmed. |
|
||||
| 20 | Scrapy vs. own framework | own | own (Scrapy considered) | **Own**. See ADR-20. |
|
||||
|
||||
## Selected ADRs
|
||||
|
||||
### ADR-1: Archive raw responses and make parsers pure
|
||||
**Context.** In v1 three parser bugs were found after a full run; fixing them required
|
||||
deleting state, rows and refetching ~1,000 pages (and politely waiting).
|
||||
**Decision.** Every fetch is logged; bodies are archived (gzip, sha256-addressed) unless the
|
||||
source forbids it (`keep_raw=False`, TDM reservation). Parsers are pure functions of the
|
||||
response. `politrack reparse <adapter>` replays the archive.
|
||||
**Consequences.** Disk usage (Parliament JSON + protocols ≈ a few GB/year; bulk sources more –
|
||||
retention configurable). Parsers become trivially unit-testable with real fixtures.
|
||||
|
||||
### ADR-2: The Loader is the only writer
|
||||
**Decision.** Adapters and enrichers yield typed records; `Loader.load()` resolves references,
|
||||
computes identity, upserts documents/claims/evidence/facets, assigns topics and refreshes
|
||||
status. Invariants (no claim without evidence, full text only where licensed, review gating)
|
||||
are enforced there.
|
||||
|
||||
### ADR-4: `identifier` table
|
||||
**Decision.** `identifier(entity_type, entity_id, scheme, value)`; schemes are free-form
|
||||
strings documented in `docs/data-model.md` (`parlament`, `wikidata`, `ep`, `bluesky`,
|
||||
`mastodon`, `telegram`, `youtube`, `uid`, `firmenbuch`, `rtr`, `eu_transparency`, …).
|
||||
|
||||
### ADR-6: Generic change detection
|
||||
**Decision.** The fetcher stores ETag/Last-Modified per request fingerprint and sends
|
||||
conditional requests; the runner skips `parse()` when the body hash and parser version equal the
|
||||
last successful parse. Adapters no longer keep "seen" dictionaries.
|
||||
|
||||
### ADR-13: Typed API, generated TS types without npm
|
||||
**Context.** The development network blocks npm; contributors should still be able to
|
||||
regenerate types. **Decision.** `politrack openapi` writes `openapi.json` and
|
||||
`web/src/lib/api-types.ts` from the FastAPI app using a small generator for the subset of
|
||||
JSON-Schema FastAPI emits.
|
||||
|
||||
### ADR-20: No Scrapy
|
||||
**Context.** Scrapy offers robots middleware, throttling, HTTP cache and pipelines.
|
||||
**Decision.** Keep a small own framework (~600 lines): our sources are mostly APIs and bulk
|
||||
files, we need DB-integrated change detection, Procrastinate scheduling, TDM handling and pure
|
||||
parsers with typed records. Scrapy's Twisted runtime and project layout would add more
|
||||
complexity than it removes. Revisit if we ever crawl large websites.
|
||||
|
||||
### ADR-21: Messaging platforms
|
||||
**Decision.** Public Telegram *channels* via their public web preview (`t.me/s/<channel>`,
|
||||
no robots restrictions, read-only, low frequency) and only for handles from verifiable
|
||||
sources (Wikidata P3789, official websites). WhatsApp Channels and Signal are not collected:
|
||||
WhatsApp offers no public read access and its terms prohibit automated collection; Signal has
|
||||
no public channels and groups are private by design.
|
||||
|
||||
### ADR-22: X only through the official API, opt-in
|
||||
**Context.** Many Austrian politicians are most active on X; proxies such as Nitter work by violating
|
||||
X's terms. **Decision.** `social.x` uses the official API v2 with the operator's bearer token and does
|
||||
nothing without one. It keeps no raw archive and no full text (`keep_raw=False`, `store_fulltext=False`).
|
||||
The Developer Agreement requires removing posts deleted on X; until a deletion sync exists the adapter is
|
||||
documented as not production-ready.
|
||||
|
||||
### ADR-23: Organisation-owned accounts
|
||||
**Context.** Party and institution channels (MFG on Telegram, NEOS and the Federal President on Bluesky)
|
||||
belong to organisations, not people. **Decision.** `social_account` and statements may belong to an
|
||||
organisation (`statement.person_id` NULL, `claim.organization_id` set); the social feed shows them, and
|
||||
party pages include the party's own channels. Curated accounts live in `config/social_accounts.yaml`
|
||||
(imported by `config.social_accounts`), each with the official page that links it.
|
||||
Reference in new issue
Block a user