politics-tracking/docs/architecture/02-comparison-and-decisions.md

Clean-room design vs. first implementation – decisions

The first implementation (v1) grew iteratively. The clean-room design (01-design-from-scratch.md) was written without looking at it. This document compares them area by area and records the decision taken for v2 (the current code). Format: lightweight ADRs.

Summary

#	Area	v1 (first implementation)	Clean-room design	Decision (v2)
1	Pipeline shape	each scraper fetches, parses and writes in one method	fetch → archive → pure parse → loader	Clean-room. Bugs found during v1 testing (vote parser, roll calls, offence regex) required resetting state and re-fetching; with an archive they are a reparse.
2	Writing to the DB	scrapers call store.record_claim(...) with hand-made fingerprint keys	typed records; Loader computes identity from resolved entity ids	Clean-room, keeping v1's battle-tested upsert SQL inside the Loader.
3	Source definition	SQL seed + scraper class + worker schedule + YAML, four places	SourceSpec in code, synced to DB; schedule on the adapter	Clean-room. One module per source.
4	External ids	columns parlament_pad, wikidata_qid	identifier(scheme, value) table	Clean-room. Needed anyway for EP, Bluesky DIDs, Telegram, company UIDs, RTR ids.
5	Entity references	scrapers look persons up themselves (SQL in scrapers)	PersonRef/OrgRef with ids/name/hints, resolved by the Loader	Clean-room. Makes parsers pure and testable.
6	Change detection	per-scraper ad-hoc signatures in scrape_state	generic: body hash + parser version per request fingerprint; conditional GET	Clean-room; v1's cursor state store kept for real cursors (dates, ids).
7	Policy enforcement	PoliteClient (robots, TDM, throttle, retries) + per-call documented_api=True	same, but exemptions declared on SourceSpec.robots_exempt	Merge: v1 client code + declarative exemptions.
8	Corroboration	post-hoc merge engine (roles, stories, events)	identity by construction + fuzzy merge pass	Both: identity by construction removes most duplicates; v1's merge engine handles near-duplicates.
9	Activity status	person_status view with verified flag and term windows	not specified in detail	v1 (proven on real data: Van der Bellen/Mikl-Leitner cases).
10	Legal matters	review queue, innocence-notice rule, official immunity data	same principles	v1 implementation, moved behind the Loader (review=True).
11	Scheduling	Procrastinate, tasks written by hand	Procrastinate, tasks generated from adapters; extra bulk/social queues	Clean-room.
12	Enrichment	functions writing claims directly	enrichers emitting records to the Loader	Clean-room.
13	API contract	handwritten dict responses; hand-maintained TS types	Pydantic response models, generated TS types	Clean-room, generated with a small built-in generator (politrack openapi --ts) so it works without npm.
14	Frontend claim rendering	one generic ClaimList with if branches	renderer registry per claim type	Clean-room (generic fallback + specific renderers).
15	Tests	inline-string unit tests	golden tests from real fixtures + unit + integration + API smoke	Clean-room, keeping v1's unit tests.
16	Data minimisation	sanitiser function in the donations scraper	enforced by record schemas	Clean-room + keep the sanitiser (defence in depth).
17	Storage of raw data	none	content-addressed raw archive, per-source keep_raw	Clean-room.
18	Plugins	none	entry points politrack.adapters	Clean-room (cheap to add, enables separate deployments).
19	Stack choices (Postgres+pgvector, FastAPI, Next.js+next-intl, MCP, Docker Compose)	✓	✓	Same in both – confirmed.
20	Scrapy vs. own framework	own	own (Scrapy considered)	Own. See ADR-20.

Selected ADRs

ADR-1: Archive raw responses and make parsers pure

Context. In v1 three parser bugs were found after a full run; fixing them required deleting state, rows and refetching ~1,000 pages (and politely waiting). Decision. Every fetch is logged; bodies are archived (gzip, sha256-addressed) unless the source forbids it (keep_raw=False, TDM reservation). Parsers are pure functions of the response. politrack reparse <adapter> replays the archive. Consequences. Disk usage (Parliament JSON + protocols ≈ a few GB/year; bulk sources more – retention configurable). Parsers become trivially unit-testable with real fixtures.

ADR-2: The Loader is the only writer

Decision. Adapters and enrichers yield typed records; Loader.load() resolves references, computes identity, upserts documents/claims/evidence/facets, assigns topics and refreshes status. Invariants (no claim without evidence, full text only where licensed, review gating) are enforced there.

ADR-4: identifier table

Decision. identifier(entity_type, entity_id, scheme, value); schemes are free-form strings documented in docs/data-model.md (parlament, wikidata, ep, bluesky, mastodon, telegram, youtube, uid, firmenbuch, rtr, eu_transparency, …).

ADR-6: Generic change detection

Decision. The fetcher stores ETag/Last-Modified per request fingerprint and sends conditional requests; the runner skips parse() when the body hash and parser version equal the last successful parse. Adapters no longer keep "seen" dictionaries.

ADR-13: Typed API, generated TS types without npm

Context. The development network blocks npm; contributors should still be able to regenerate types. Decision. politrack openapi writes openapi.json and web/src/lib/api-types.ts from the FastAPI app using a small generator for the subset of JSON-Schema FastAPI emits.

ADR-20: No Scrapy

Context. Scrapy offers robots middleware, throttling, HTTP cache and pipelines. Decision. Keep a small own framework (~600 lines): our sources are mostly APIs and bulk files, we need DB-integrated change detection, Procrastinate scheduling, TDM handling and pure parsers with typed records. Scrapy's Twisted runtime and project layout would add more complexity than it removes. Revisit if we ever crawl large websites.

ADR-21: Messaging platforms

Decision. Public Telegram channels via their public web preview (t.me/s/<channel>, no robots restrictions, read-only, low frequency) and only for handles from verifiable sources (Wikidata P3789, official websites). WhatsApp Channels and Signal are not collected: WhatsApp offers no public read access and its terms prohibit automated collection; Signal has no public channels and groups are private by design.

ADR-22: X only through the official API, opt-in

Context. Many Austrian politicians are most active on X; proxies such as Nitter work by violating X's terms. Decision. social.x uses the official API v2 with the operator's bearer token and does nothing without one. It keeps no raw archive and no full text (keep_raw=False, store_fulltext=False). The Developer Agreement requires removing posts deleted on X; until a deletion sync exists the adapter is documented as not production-ready.

ADR-23: Organisation-owned accounts

Context. Party and institution channels (MFG on Telegram, NEOS and the Federal President on Bluesky) belong to organisations, not people. Decision. social_account and statements may belong to an organisation (statement.person_id NULL, claim.organization_id set); the social feed shows them, and party pages include the party's own channels. Curated accounts live in config/social_accounts.yaml (imported by config.social_accounts), each with the official page that links it.