simonandClaude Opus 5.5
693c32a5e9
social.x_deletions checks stored posts through the official API. A post X reports as not found keeps a marked record (author, link, dates, Internet Archive link if a copy exists) while every copy of its text is removed, as the Developer Agreement requires. A post that shows up in the timeline again is restored by the normal upsert. mcp 2.x removed FastMCP, which crash-looped the MCP server on first deploy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
94 lines
7.9 KiB
Markdown
94 lines
7.9 KiB
Markdown
# Clean-room design vs. first implementation – decisions
|
||
|
||
The first implementation (v1) grew iteratively. The clean-room design
|
||
([01-design-from-scratch.md](01-design-from-scratch.md)) was written without looking at it.
|
||
This document compares them area by area and records the decision taken for v2 (the current
|
||
code). Format: lightweight ADRs.
|
||
|
||
## Summary
|
||
|
||
| # | Area | v1 (first implementation) | Clean-room design | **Decision (v2)** |
|
||
|---|---|---|---|---|
|
||
| 1 | Pipeline shape | each scraper fetches, parses and writes in one method | fetch → archive → pure parse → loader | **Clean-room**. Bugs found during v1 testing (vote parser, roll calls, offence regex) required resetting state and re-fetching; with an archive they are a `reparse`. |
|
||
| 2 | Writing to the DB | scrapers call `store.record_claim(...)` with hand-made fingerprint keys | typed records; Loader computes identity from resolved entity ids | **Clean-room**, keeping v1's battle-tested upsert SQL inside the Loader. |
|
||
| 3 | Source definition | SQL seed + scraper class + worker schedule + YAML, four places | `SourceSpec` in code, synced to DB; schedule on the adapter | **Clean-room**. One module per source. |
|
||
| 4 | External ids | columns `parlament_pad`, `wikidata_qid` | `identifier(scheme, value)` table | **Clean-room**. Needed anyway for EP, Bluesky DIDs, Telegram, company UIDs, RTR ids. |
|
||
| 5 | Entity references | scrapers look persons up themselves (SQL in scrapers) | `PersonRef`/`OrgRef` with ids/name/hints, resolved by the Loader | **Clean-room**. Makes parsers pure and testable. |
|
||
| 6 | Change detection | per-scraper ad-hoc signatures in `scrape_state` | generic: body hash + parser version per request fingerprint; conditional GET | **Clean-room**; v1's cursor state store kept for real cursors (dates, ids). |
|
||
| 7 | Policy enforcement | `PoliteClient` (robots, TDM, throttle, retries) + per-call `documented_api=True` | same, but exemptions declared on `SourceSpec.robots_exempt` | **Merge**: v1 client code + declarative exemptions. |
|
||
| 8 | Corroboration | post-hoc merge engine (roles, stories, events) | identity by construction + fuzzy merge pass | **Both**: identity by construction removes most duplicates; v1's merge engine handles near-duplicates. |
|
||
| 9 | Activity status | `person_status` view with verified flag and term windows | not specified in detail | **v1** (proven on real data: Van der Bellen/Mikl-Leitner cases). |
|
||
| 10 | Legal matters | review queue, innocence-notice rule, official immunity data | same principles | **v1** implementation, moved behind the Loader (`review=True`). |
|
||
| 11 | Scheduling | Procrastinate, tasks written by hand | Procrastinate, tasks generated from adapters; extra `bulk`/`social` queues | **Clean-room**. |
|
||
| 12 | Enrichment | functions writing claims directly | enrichers emitting records to the Loader | **Clean-room**. |
|
||
| 13 | API contract | handwritten dict responses; hand-maintained TS types | Pydantic response models, generated TS types | **Clean-room**, generated with a small built-in generator (`politrack openapi --ts`) so it works without npm. |
|
||
| 14 | Frontend claim rendering | one generic `ClaimList` with `if` branches | renderer registry per claim type | **Clean-room** (generic fallback + specific renderers). |
|
||
| 15 | Tests | inline-string unit tests | golden tests from real fixtures + unit + integration + API smoke | **Clean-room**, keeping v1's unit tests. |
|
||
| 16 | Data minimisation | sanitiser function in the donations scraper | enforced by record schemas | **Clean-room** + keep the sanitiser (defence in depth). |
|
||
| 17 | Storage of raw data | none | content-addressed raw archive, per-source `keep_raw` | **Clean-room**. |
|
||
| 18 | Plugins | none | entry points `politrack.adapters` | **Clean-room** (cheap to add, enables separate deployments). |
|
||
| 19 | Stack choices (Postgres+pgvector, FastAPI, Next.js+next-intl, MCP, Docker Compose) | ✓ | ✓ | **Same in both** – confirmed. |
|
||
| 20 | Scrapy vs. own framework | own | own (Scrapy considered) | **Own**. See ADR-20. |
|
||
|
||
## Selected ADRs
|
||
|
||
### ADR-1: Archive raw responses and make parsers pure
|
||
**Context.** In v1 three parser bugs were found after a full run; fixing them required
|
||
deleting state, rows and refetching ~1,000 pages (and politely waiting).
|
||
**Decision.** Every fetch is logged; bodies are archived (gzip, sha256-addressed) unless the
|
||
source forbids it (`keep_raw=False`, TDM reservation). Parsers are pure functions of the
|
||
response. `politrack reparse <adapter>` replays the archive.
|
||
**Consequences.** Disk usage (Parliament JSON + protocols ≈ a few GB/year; bulk sources more –
|
||
retention configurable). Parsers become trivially unit-testable with real fixtures.
|
||
|
||
### ADR-2: The Loader is the only writer
|
||
**Decision.** Adapters and enrichers yield typed records; `Loader.load()` resolves references,
|
||
computes identity, upserts documents/claims/evidence/facets, assigns topics and refreshes
|
||
status. Invariants (no claim without evidence, full text only where licensed, review gating)
|
||
are enforced there.
|
||
|
||
### ADR-4: `identifier` table
|
||
**Decision.** `identifier(entity_type, entity_id, scheme, value)`; schemes are free-form
|
||
strings documented in `docs/data-model.md` (`parlament`, `wikidata`, `ep`, `bluesky`,
|
||
`mastodon`, `telegram`, `youtube`, `uid`, `firmenbuch`, `rtr`, `eu_transparency`, …).
|
||
|
||
### ADR-6: Generic change detection
|
||
**Decision.** The fetcher stores ETag/Last-Modified per request fingerprint and sends
|
||
conditional requests; the runner skips `parse()` when the body hash and parser version equal the
|
||
last successful parse. Adapters no longer keep "seen" dictionaries.
|
||
|
||
### ADR-13: Typed API, generated TS types without npm
|
||
**Context.** The development network blocks npm; contributors should still be able to
|
||
regenerate types. **Decision.** `politrack openapi` writes `openapi.json` and
|
||
`web/src/lib/api-types.ts` from the FastAPI app using a small generator for the subset of
|
||
JSON-Schema FastAPI emits.
|
||
|
||
### ADR-20: No Scrapy
|
||
**Context.** Scrapy offers robots middleware, throttling, HTTP cache and pipelines.
|
||
**Decision.** Keep a small own framework (~600 lines): our sources are mostly APIs and bulk
|
||
files, we need DB-integrated change detection, Procrastinate scheduling, TDM handling and pure
|
||
parsers with typed records. Scrapy's Twisted runtime and project layout would add more
|
||
complexity than it removes. Revisit if we ever crawl large websites.
|
||
|
||
### ADR-21: Messaging platforms
|
||
**Decision.** Public Telegram *channels* via their public web preview (`t.me/s/<channel>`,
|
||
no robots restrictions, read-only, low frequency) and only for handles from verifiable
|
||
sources (Wikidata P3789, official websites). WhatsApp Channels and Signal are not collected:
|
||
WhatsApp offers no public read access and its terms prohibit automated collection; Signal has
|
||
no public channels and groups are private by design.
|
||
|
||
### ADR-22: X only through the official API, opt-in
|
||
**Context.** Many Austrian politicians are most active on X; proxies such as Nitter work by violating
|
||
X's terms. **Decision.** `social.x` uses the official API v2 with the operator's bearer token and does
|
||
nothing without one. It keeps no raw archive and no full text (`keep_raw=False`, `store_fulltext=False`).
|
||
The Developer Agreement requires removing posts deleted on X. For accountability we mark instead of erase:
|
||
`social.x_deletions` clears every copy of the text and keeps a record that the post existed and disappeared
|
||
(a new `PostRemoved` record, so the Loader stays the only writer).
|
||
|
||
### ADR-23: Organisation-owned accounts
|
||
**Context.** Party and institution channels (MFG on Telegram, NEOS and the Federal President on Bluesky)
|
||
belong to organisations, not people. **Decision.** `social_account` and statements may belong to an
|
||
organisation (`statement.person_id` NULL, `claim.organization_id` set); the social feed shows them, and
|
||
party pages include the party's own channels. Curated accounts live in `config/social_accounts.yaml`
|
||
(imported by `config.social_accounts`), each with the official page that links it.
|