simonandClaude Opus 5.5
456f9816f4
Design concept and design system in docs/design/: principles (evidence
first, neutral, fair to the people shown, plain language, time as a
dimension, accessible and private), information architecture, core
patterns and a concept for every page; tokens with checked contrast,
component catalogue, content and accessibility rules.
web:
- neutral ink-indigo accent instead of flag red; party colours only as
small markers next to the name; light/dark theme with a switcher
- stylesheets split into tokens, base, layout, components, print
- new header (search, filter count, language, theme, mobile menu) and
footer; breadcrumbs, page headers, tabs with counts, filter bars that
collapse on phones, pagination, notices and empty states
- claim cards with source meter, legal stage, innocence notice inside
the card and a permalink; claim pages with citation (/angaben/{id})
- profiles with role timeline and key figures; party and organisation
dashboards; vote list with position chips; calendar agenda on phones;
tables that stack into cards; loading and error pages; favicon
backend:
- GET /api/v1/claims/{id} and MCP tool get_claim for permalinks
- parties list: evaluate person_status once instead of per party
(30 s -> 1 s on the sample database, identical result)
- parliament calendar: HTML-decode titles ("84. Sitzung")
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
69 lines
4.9 KiB
Markdown
69 lines
4.9 KiB
Markdown
# Architecture (overview)
|
||
|
||
This page is the short version. The full design – quality goals, data model, the adapter framework down
|
||
to the request level, testing and guard-rails – is in
|
||
[architecture/01-design-from-scratch.md](architecture/01-design-from-scratch.md); the reasoning behind
|
||
each decision is in [architecture/02-comparison-and-decisions.md](architecture/02-comparison-and-decisions.md).
|
||
|
||
## Deployment
|
||
|
||
```
|
||
┌──────────────────────────── Unraid host ─────────────────────────────┐
|
||
Browser ──► reverse │ web (Next.js, de/en) ──► api (FastAPI, typed JSON) ──┐ │
|
||
(SWAG/NPM/Caddy) │ ▼ │
|
||
AI agents ─────────► │ mcp (MCP server, streamable HTTP) ──────────► PostgreSQL 16 │
|
||
BI / notebooks ────► │ (read-only role politrack_reader) ──────────► + pgvector, pg_trgm │
|
||
│ + procrastinate queue │
|
||
│ worker-scrape queues scrape,social (N replicas) ──┤ │
|
||
│ worker-bulk queue bulk (large downloads) ───────┤ │
|
||
│ worker-enrich queue enrich (linking, NLP, corroboration) ─┘ │
|
||
│ /data/raw raw-response archive (gzip, content-addressed) │
|
||
│ ollama (optional, bge-m3 embeddings) │
|
||
└────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
One backend image runs every role (API, MCP, workers, CLI). Workers can run on other machines that reach
|
||
the database (e.g. a scraper box with its own egress IP, a GPU box for embeddings).
|
||
|
||
## Pipeline in one picture
|
||
|
||
```
|
||
SourceSpec + Adapter.start() ─▶ Fetcher ─▶ raw archive ─▶ Adapter.parse() ─▶ Records ─▶ Loader ─▶ PostgreSQL
|
||
(code) robots.txt, throttle, (pure) typed resolve, ▲
|
||
TDM, conditional GET identity, │
|
||
evidence │
|
||
Enrichers (linker, events, legal, topics, corroborate) ───┘
|
||
```
|
||
|
||
* **Four strict stages** – fetch (policy + archive), parse (pure), load (single writer), enrich.
|
||
* **Everything is a claim with evidence**; the same fact from two sources is one claim (corroboration by
|
||
construction).
|
||
* **Sources are code** (`SourceSpec` next to the adapter), synced into the `source` table.
|
||
* **Re-parse without re-fetch** (`politrack reparse <adapter>`) after parser fixes.
|
||
* **Typed API** (Pydantic) → OpenAPI → generated TypeScript types for the web app.
|
||
|
||
## Why this stack
|
||
|
||
| Concern | Choice | Reason |
|
||
|---|---|---|
|
||
| Database | PostgreSQL 16 + pgvector + pg_trgm + unaccent | relational data, German full-text search, fuzzy names and vectors in one easily backed-up system that agents can query |
|
||
| Job queue | Procrastinate (Postgres-backed) | no Redis; cron-style periodic tasks, retries, queueing locks; horizontal workers |
|
||
| Backend | Python 3.12, httpx, protego, psycopg 3, FastAPI | best scraping/NLP ecosystem; one code base for collection and serving |
|
||
| Frontend | Next.js 15 + next-intl | server-side rendering, German at `/`, English at `/en` |
|
||
| Agents / RAG | MCP server, pgvector chunks, `COMMENT ON` everywhere, read-only SQL role | tools or plain SQL; every answer carries citation URLs |
|
||
| Embeddings | Ollama + bge-m3 (optional) | local, multilingual; full-text search works without it |
|
||
|
||
No Scrapy: its strengths (crawling at scale) are not our problem, while its model fights ours (pure
|
||
parsers, archive-first, per-publisher legal policy) – see ADR-20.
|
||
|
||
## Interfaces for other agents
|
||
|
||
1. **MCP** – `http://<host>:8001/mcp`: `search`, `find_person`, `get_person`, `person_network`,
|
||
`list_parties`, `get_party`, `find_organization`, `get_organization`, `list_claims`, `get_claim`, `money_flows`,
|
||
`social_posts`, `wikipedia_edits`, `calendar`, `person_votes`, `former_politicians`, `list_topics`.
|
||
2. **REST** – `http://<host>:8000/api/docs` (OpenAPI 3).
|
||
3. **SQL** – role `politrack_reader`; start with the view `claim_with_evidence`; `\d+ claim` shows the
|
||
column comments.
|
||
4. **RAG** – table `chunk` (text, `tsv`, `embedding vector(1024)`), each chunk linked to a claim or
|
||
document for citations.
|